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FOREWORD 


This  report  presents  the  final  results  in  the  design  of  an  Educational 
Technology  Assessment  Model  (ETAM),  a training  management  tool  for  assess- 
ing the  long-term  effects  of  introducing  an  innovative  technological  change 
into  the  existing  or  planned  training  environment.  This  represents  the 
final  output  of  Phase  II-A  which  is  part  of  a multiphase  effort  called 
"Design  of  Training  Systems,"  undertaken  in  consonance  with  the  requirements 
of  Advanced  Development  Objective  43-03X,  "Education  and  Training." 

Sincere  thanks  is  expressed  for  the  cooperation  of  the  Director  and  members 
of  the  Training  Analysis  and  Evaluation  Group  for  making  available  certain 
documentation  and  in  providing  assistance  in  various  areas  of  the  project. 
Especially  helpful  was  the  TECEP  report,  which  included  training  cost  model 
algorithms,  made  available  by  TAEG.  The  assistance  provided  by  Dr.  William 
Giauque,  Naval  Postgraduate  School,  was  extremely  helpful  in  finalizing  the 
decision  analytic  components  of  the  procedure.  Special  acknowledgement  is 
given  Prof.  Robert  M.  Gagne,  Florida  State  University,  who  served  as  project 
consul tant. 

The  ETAM  procedural  approach  was  developed  by  Dr.  R.  B.  Miller  and  Mr.  L.  R. 
Duffy.  Mr.  R.  E.  Hallman  provided  management  for  the  project;  Mrs.  L.  Girard, 
Mrs.  C.  Latham  and  Mrs.  S.  Goodell  provided  secretarial  and  editorial  services. 
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SECTION  I 

EDUCATIONAL  TECHNOLOGY  ASSESSMENT  MODEL  OVERVIEW 


PROBLEM 

Rapid  changes  are  coming  about  in  the  demands  put  on  education  and  training, 
but  there  are  also  extensions  being  made  continuously  to  educational  and  train- 
ing technology.  The  decision  to  adopt,  reject,  postpone,  or  further  study  a 
proposed  innovation  should  be  a rational  choice  based  upon  the  relationship  of 
the  costs  to  introduce  and  maintain  the  innovation,  balanced  against  the  range 
and  magnitude  of  the  benefits  to  be  derived  from  the  innovation.  It  is  im- 
portant to  identify,  as  precisely  as  possible,  the  range  of  applicability  of 
the  Innovation  in  order  to  appreciate  the  economies  of  scale  in  realizing  the 
benefits,  while  avoiding  the  costs  of  error  in  misapplying  the  innovation  to 
training  situations  and  other  conditions  where  the  innovation  is  irrevelant 
or  contra-indicated. 

The  training  enterprise  is  a system  of  interlocking  activities  where  a given 
change  may  be  reflected  in  a large  pattern  of  consequences.  Thus,  if  something 
is  made  "easier  to  learn,"  the  outcome  can  be  a reduction  in  the  aptitude  re- 
quirements of  the  student,  a reduction  of  training  time,  or  a higher  level  of 
capability  of  the  trained  student  for  a given  amount  of  training  time.  The 
choice  of  which  variable  to  exploit  as  an  advantage  can  depend  on  the  relative 
cost  advantage  to  the  personnel  subsystem  in  terms  of  available  recruits,  or 
to  the  training  establishment  in  terms  of  costs  per  graduating  student,  or  to 
operations  in  terms  of  skill  level.  The  assessment  technique  leading  to 
decisions  should  enable  the  analysis  leading  to  these  choices. 

Instructional  technology  embraces  a wide  variety  of  elements.  These  range  from 
costs  and  capabilities  of  dozens  of  types  of  vehicles,  from  human  instructor  and 
blackboard  to  flight  simulator.  The  technology  also  applies  to  instructional 
techniques  and  learning  controls,  student  incentives  and  motivations,  transfer 
of  training  facilitations,  methods  of  providing  practice  in  knowledge  and  skill 
learning,  the  organization  of  instructional  sequence  and  content,  and  a host  of 
other  factors  In  the  efficiency  and  effectiveness  of  the  teaching-learning 
process.  Instructional  technology  also  extends  to  the  technical  and  administra- 
tive operations  by  which  job  task  requirements  are  Identified,  training 
objectives  specified,  Instructional  vehicles  are  programmed  for  instruction, 
and  student  performances  measured  and  interpreted.  All  of  these  factors  inter- 
act with  benefits  and  costs  associated  with  where  specific  training  will  be 
conducted,  when  it  will  be  taught,  and  under  what  conditions.  The  total  benefit 
and  cost  model  to  which  a substantive  innovation  is  to  be  applied  is  very  complex. 

At  this  point  the  reader  may  have  only  a vague  concept  for  the  term  "innovation" 
as  used  In  the  ETAM  context.  Early  in  Section  III,  characteristics  and  examples 
of  training  innovations  are  discussed  in  detail.  However,  here  it  should  be 
sufficient  to  state  that  innovation  is  a constant  or  enduring  change  that  can 
be  purposely  planned  for  and  evaluated  on  the  basis  of  known  criteria.  Thus, 
while  "change"  may  be  the  unplanned  evolution  from  one  state  to  another,  innova- 
tion implies  that  the  state  change  can  and  is  planned  and  evaluated. 


TAEG  REPORT  NO.  12-3 


But  the  introduction  of  an  innovation  can  have  remote  effects  which  cannot  be 
readily  anticipated.  Its  outcomes  are  only  partly  predictable,  even  by  knowl- 
edgeable experts,  so  that  the  decision  to  accept  or  reject  an  innovation  at  a 
given  time  is  based  on  probabilities  of  success  and  perceived  magnitudes  of 
risk.  Rational  choices  should  objectify,  as  far  as  possible,  these  uncertain- 
ties, probabilities,  and  risks  when  the  potential  benefits  and  costs  of  the 
innovation  are  compared  with  alternatives.  One  alternative  is  not  to  make  the 
change  but  continue  as  is  with  respect  to  the  proposed  innovation.  But  the 
risks  of  this  conservative  policy  should  also  be  made  explicit. 

The  objective  of  this  procedure  for  evaluating  innovations  or  changes  proposed 
for  the  instructional  enterprise  is  to  formalize  as  extensively  as  possible, 
the  dimensions  and  elements  of  potential  applicability  of  the  innovation,  and 
to  structure  the  benefit  variables  and  cost  variables  that  are  relevant  to  it. 

The  outcome  should  be  a presentation  to  the  executive  decision  maker  of  the 
profile  of  expected  pattern  of  potential  benefits  and  cost  advantages,  the 
estimated  probabilities  of  achieving  the  advantages,  the  major  liabilities  that 
confront  their  achievement,  and,  where  choice  or  rejection  may  not  be  clear-cut, 
the  most  critical  data  to  next  obtain  for  making  that  decision.  The  decision 
maker  may  then  be  in  a position  to  evaluate  a variety  of  innovation  proposals 
that  compete  for  an  inevitably  limited  budget,  so  that  maximum  returns  on  in- 
vestment can  be  realized. 

A secondary  and  dependent  objective  is  to  develop  a descriptive  and  analytic 
terminology  that  can  be  applied  to  data  base  descriptions  of  instructional 
vehicles  and  their  relevant  properties,  job  tasks  and  their  important  attri- 
butes, and  training  courses  and  content  within  training  courses.  By  attaching 
relevant  descriptors  to  identify  the  entire  domain  of  variables  in  the  training 
enterprise,  it  becomes  possible  to  use  the  computer  for  making  automatic  searches 
for  relevant  entities  in  courses,  and  in  instructional  vehicles,  after  an  innova- 
tion has  been  described  by  appropriate  descriptors  from  the  descriptor  list. 

These  descriptor  sets  may  also  structure  cost  models  for  categorizing  and  access- 
ing empirical  cost  data.  These  objectives  do  not  assume  that  the  final  decision 
to  accept  or  reject  will  be  based  solely  upon  the  items  turned  up  by  computer 
search;  rather,  all  ultimate  decisions  of  this  kind  will  be  made  by  human  experts. 

APPROACH 

The  entire  ETAM  procedure  is  aimed  at  supporting  human  judgment  and  reducing  or 
simplifying  the  exercise  of  human  judgment,  but  not  of  replacing  it  by  automatic 
operations  of  selection  and  calculation.  The  large  number  of  loose  interactions 
In  the  development  and  administration  of  training,  in  the  learning  process,  and 
between  the  learning  process  and  the  characteristics  of  the  student,  demand  in- 
formed human  judgmental  processes  for  decisions  and  outcome  monitoring.  The 
procedures  are  intended  to  extend  and  support  human  judgments  by  offering 
structure,  reference  definitions,  and  concepts,  and  in  general,  by  trying  to 
get  the  most  use  of  information  available  to  the  assessor,  either  within  his 
head  or  external  to  it. 

Even  where  a set  of  automatic  operations  combines  and  integrates  assessment  data 
in  executing  the  algorithms  of  the  assessment  model,  the  constituent  data  are 
also  displayed  to  the  decision  maker  for  his  inspection.  There  is  every  oppor- 
tunity for  making  commonsense  checks  against  "black  box"  operations. 
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The  procedural  approach  has  also  aimed  at  maximum  consistency  with  Navy  concepts 
and  terminology,  compatible  with  the  practical  requirements  of  making  the  assess- 
ment. It  has  been  necessary,  however,  to  formulate  an  organized  picture  of  in- 
structional functions,  of  the  learning  process  as  related  to  defined  task  struc- 
tures, and  of  generalized  differences  in  instructional  requirements  for  different 
classes  of  job  tasks.  The  intent  in  creating  these  taxonomic  structures  has 
been  to  compromise  between  academic  precision  and  practical  simplicity.  Every 
term  is  defined  at  one  or  more  levels  in  readily  identified  contexts. 

It  must,  therefore,  be  stipulated  that  the  use  of  ETAM  in  the  assessment  of 
innovations  requires  persons  with  extensive  background  knowledge  in  Navy  train- 
ing operations,  who  have  had  at  least  a number  of  days  study  of  the  ETAM  pro- 
cedures and  the  referenced  material  which  defines  the  application  of  the 
terminology. 

An  attempt  has  been  made  to  safeguard  and  offset  both  of  the  following  types 
of  errors  from  being  grossly  manifest: 

a.  Accepting  an  innovation  because  its  benefits  have  been  overgeneralized, 
or  its  costs  and  liabilities  have  been  incompletely  recognized. 

b.  Rejecting  an  innovation  because  the  range  of  its  benefits  has  been  too 
narrowly  perceived,  or  because  its  costs  have  been  too  narrowly  prorated, 
or  because  its  liabilities  have  not  been  presented  in  ways  that  enable 
their  overcoming  to  be  feasible. 

CONCLUSIONS 

ETAM  exists  as  a preliminary  design  and,  as  such,  it  has  been  concluded  that: 

• A systematic  procedural  methodology  for  assessing  innovation  and  change 
in  training  can  be  described. 

t The  following  of  a procedural  model  can  lead  to  a more  effective  analysis 
of  a proposed  innovation. 

• A decision  structure  can  be  organized  that  permits  a systematic  assess- 
ment of  both  tangible  and  intangible  benefits. 

• A decision  sequence  can  be  developed  which  makes  effective  use  of 
efficiently  gathered  information. 

• A taxonomic  structure  can  be  used  strategically  to  search  out  the  full 
range  of  application  of  any  innovation  or  change  within  the  instruc- 
tional enterprise. 

• ETAM,  as  a totally  manual  procedure,  demands  an  impractical  amount  of 
time  and  effort  for  exploring  multiple  alternatives,  variations,  and 
conditions. 

• However,  many  components  of  ETAM  demand  human  (expert)  judgments  which 
can  be  supported  but  not  replaced  with  automated  procedures. 
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RECOMMENDATIONS 

It  was  concluded  that  the  ETAM  design  provides  a logical  basis  for  evaluating 
an  innovation;  however,  the  manual  aspects  of  the  procedure  cause  it  to  be  un- 
workable for  anything  beyond  an  analysis  involving  just  a few  alternatives, 
variations,  and  conditions.  It  is  recommended  that  automated  procedures  be 
developed  to  facilitate  more  complex  analyses  and  that  this  development  effort 
be  supported  to  include  the  following: 

• A study  of  the  scaling  procedures  for  expressing  the  relative  "importance" 
of  benefits  and  the  probabilities  of  success,  as  a refinement  of  the  cost- 
benefits  assessment  process. 

• Development  of  the  indexing  procedures  for  translating  the  ETAM 
nomenclature  into  descriptions  of  courses,  tasks,  and  instructional 
vehicle  attributes.  This  would  include  translation  rules  and  guide- 
lines for  assigning  data  base  search  descriptors. 

• A study  of  the  human  factors  requirements  and  the  development  of  a 
procedural  interface  for  an  assessor  to  interact  with  a graphic 
terminal  linked  to  a computer  in  performing  information  retrieval, 
contextual  manipulation  of  decision  data,  and  user  data  entry. 

• The  specification  of  an  information  flow  model  for  the  functions  out- 
lined in  the  Development  and  Adnini strati ve  Management  of  Training 
(Section  III,  Subtask  05.24),  to  produce  a more  tightly  coupled  system 
for  predicting  training/ job  effects  from  changes  in  these  functions. 

• Development  of  a series  of  integrated  programs,  based  upon  the  computer 
applications  outlined  in  Section  IV,  which  will  accept  user  generated 
input,  interact  with  the  user  to  obtain  refined  data,  and  will  provide 
a set  of  output  reports  to  aid  the  decision-making  process. 
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SECTION  II 

REVIEW  OF  RELEVANT  LITERATURE 


REVIEW  OF  THE  LITERATURE  ON  TASK  TAXONOMIES  AND  DECISION  ANALYSIS 

The  taxonomic  structures  offered  in  Task  5 in  the  ETAM  procedures  deal  with 
the  determination  of  the  potential  range-of-effect  in  a proposed  innovation. 

The  classifications  in  that  section  are  based  mainly  on  a sequence  of  develop- 
ments by  R.  B.  Miller  that  began  in  the  early  1 950 *s  with  his  outline  for  the 
description  and  analysis  of  military  tasks,  primarily  in  the  operation  and 
maintenance  of  equipments.  These  techniques  were  best  organized  and  presented 
in  his  1963  article.  This  article  also  contained  the  classification  of  task 
elements  or  task  functions  which  are  substantially  retained  in  ETAM.  The 
structure  for  Stages  of  Learning  was  contained  in  his  Handbook  on  Training 
and  Training  Equipment  Design  (1953),  which  was  extended  to  include  the  con- 
cepts of  Task  Formats  in  his  monograph  on  A Method  for  Determining  Task 
Strategies  (1974).  His  1972  publication  on  the  design  of  Duty  Modules  pro- 
poses  a systematic  method  for  the  organizing  of  tasks  into  entities  useful  for 
training,  operations,  skills  descriptions,  and  transfer  of  training  or  assign- 
ment purposes. 

The  literature  on  task  taxonomies  tends  to  be  more  extensive  than  illuminating, 
if  one  is  concerned  with  practical  utility.  It  is  outside  the  scope  of  rele- 
vance here  to  review  the  range  of  approaches,  arguments,  and  findings.  Miller's 
1975  paper  contains  such  a general  review  with  critiques.  The  same  document 
which  contains  Miller's  review,  also  contains  the  taxonomic  positions  of  Gagne 
(1975)  and  Fleishman  (1975),  respectively.  All  three  articles  point  to  exten- 
sive reference  bibliographies  on  taxonomic  questions. 

In  summary,  taxonomic  positions  and  aspirations  have  been  based  on  three  quite 
different  starting  assumptions.  Miller  summarized  two  of  these  three  positions 
in  1967.  One  major  school  of  taxonomic  development  is  dedicated  to  the  empirical 
discovery  of  taxonomic  elements  in  behavior  or  performance.  (Some  theorists 
hold  that  behavior  should  be  treated  differently  in  this  context  than  performance, 
others  hold  that  the  distinction  is  meaningless  at  best,  misleading  at  worst.) 
Fleishman  is  the  most  systematic  and  articulate  holder  of  this  position.  It  is 
believed  that  whatever  performance  components  are  revealed,  essentially  by 
factor  analytic  studies,  are  structural  entities  (abilities)  which  are  inde- 
pendent of  the  instructional  technique  or  other  influences  on  the  learner's 
experiences  which  become  manifest  in  a skill. 

According  to  Miller  (1967,  1 971 A , 1971B),  the  second  point  of  view  is  that 
behavioral  "functions"  are  inevitably  ambiguous  in  their  definition  and  speci- 
fication except  in  the  denotable  instance;  but  this  is  not  a taxonomic  occa- 
sion which  should  be  a generalization  of  instances.  Furthermore,  Miller 
insists,  a large  variety  of  quite  different  learning  experiences  may  become 
associated  in  the  expression  of  a skilled  performance.  He  believes,  there- 
fore, that  task  taxonomies  are  invented  and  have  as  their  criterion  not  some 
natural  structures  that  can  be  independently  validated,  but  that  their  test 
is  in  practical  utility.  Thus,  it  is  useful  to  distinguish  between  something 
called  "scan  and  detect"  and  something  called  "interpret"  because  one  can  be 
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efficiently  learned  by  different  presentations  and  experiences  than  the  other, 
and  each  generalizes  differently  than  the  other.  Miller  would  say  that  if  a 
categorization  does  not  serve  a useful  design  or  decision  purpose,  it  should 
be  abandoned  for  one  that  does.  In  this  position,  he  treats  task  taxonomies 
as  a branch  of  technology,  whereas  Fleishman  (and  many  of  his  colleagues) 
treats  task  taxonomy  development  as  a branch  of  science. 

GagnS  represents  a third  position.  In  his  Conditions  of  Learning  (1970),  he 
confines  his  interest  to  cognitive  tasks.  He  identifies  layers  of  building- 
block  learning  elements,  where  a more  complex  level  of  competence  is  achieved 
by  putting  together  simpler  building  blocks,  which  are  conceived  as  prerequi- 
sites. He  thus  distinguishes  categories  of  human  performance  which,  on  the  one 
hand,  imply  the  existence  of  different  human  capabilities,  and  on  the  other,  re- 
quire different  sets  of  conditions  for  their  learning.  In  later  work  (Gagn§ 
and  Briggs,  1974),  this  point  of  view  is  expanded  to  encompass  additional  cate- 
gories of  human  competence  including  knowledge  and  motor  skills. 

The  literature  on  task  analysis  is  largely  methodological  and  intersects  little 
with  theory  and  concepts  of  practice  in  learning  and  behavior  organization. 

Miller  (1962)  has  proposed  a differentiation  between  task  description,  which  is 
an  objective  statement  of  what  a performance  should  do,  and  task  analysis,  which 
deals  primarily  with  the  psychological  conditions  for  effective  performance. 

In  its  broad  application,  task  analyses  of  different  kinds  support  varied  pur- 
poses in  the  personnel  subsystem:  job  and  task  design,  human  factors  engineer- 

ing, job  aids,  training,  assessment  of  skills,  skills  identification,  assign- 
ment, and  selection.  Task  descriptions  appropriate  for  one  objective  may  be 
inadequate  for  other  of  these  purposes.  Task  descriptions,  as  the  basis  for 
the  development  of  training  content,  have  become  almost  standard  doctrine  in 
the  Navy,  as  well  as  the  Air  Force  and  Army.  A variety  of  formats  have  been 
proposed,  but  most  of  them  follow  the  structure  of  "stimulus  situation-action- 
consequence  of  action"  as  the  paradigm  for  the  unit  of  description.  In  general, 
however,  formats  for  capturing  problem  solving  activities,  such  as  fault  localiza- 
tion, have  been  inadequate  as  direct  input  to  instructional  design.  Miller  (1974) 
has  attempted  to  remedy  some  of  these  inadequacies. 


Rundquist  (1970)  has  laid  out  the  most  practical  and  comprehensive  pattern  of 
procedural  steps  known  to  this  reviewer  for  describing  duties  and  job  tasks, 
and  for  proceeding  from  these  descriptions  to  the  specifications  of  courses 
and  detailed  course  objectives.  It  is  especially  effective  in  topics  about 
which  many  guides  are  deficient;  i.e.,  the  determination  of  the  range  of 
input  situations  and  conditions  with  which  the  job  incumbent  must  cope,  and, 
therefore,  to  which  he  should  be  exposed,  on  an  appropriate  sampling  basis, 
during  training.  The  guidance  contains  many  insights  about  the  psychology 
of  learning  that  should  be  applied  to  the  instructional  process.  While  it 
is  procedurally  comprehensive,  both  in  range  and  depth,  for  developing  and 
converting  job  information  into  training  specifications  and  operations,  the 
guide  is  not  concerned  with  taxonomic  issues,  and  indeed  they  are  irrelevant 
to  its  purposes.  Rundquist's  procedures  are  being  widely  applied  In  the 
Navy.  This  job-skill  oriented  approach  to  the  identification  and  specification 
of  training  objectives  should  facilitate  the  application  of  the  ETAM  taxonomies. 
At  present,  this  seems  a reasonable  conjecture  rather  than  a confirmed 
demonstration. 
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An  excellent  reference  to  a complete  list  of  types  and  training  devices  is 
contained  in  a preliminary  U.  S.  Navy  Training  Analysis  and  Evaluation  Group 
report  entitled,  "A  Technique  for  Choosing  Cost-Effective  Instructional  Media." 
An  extremely  useful  set  of  guidelines  for  the  choice  of  training  equipment  and 
media  applications  is  also  contained  in  this  report. 

The  literature  on  decision  theory  has  grown  extensively  over  the  past  few  years. 
A major  part  of  this  literature  deals  with  a "decision  science"  approach  to 
combining  subjective  probabilities  and  multi-dimensional  utility  patterns. 
Schlaifer  (1959)  introduces  some  of  the  basic  concepts  for  obtaining  the  ex- 
pected value  of  utility  patterns.  Rather  separate  components  of  the  decision 
process  are  being  combined  to  bring  more  sophisticated  methods  to  bear  upon 
the  solution  of  complex  problems.  A Handbook  for  Decision  Analysis  (1973), 
developed  as  an  outgrowth  of  a contract  with  the  Office  of  Naval  Research  (ONR), 
treats  a number  of  the  major  concepts  and  techniques  in  decision  analysis.  It 
contains  numerous  examples  of  their  application  to  practical  problems.  The 
basis  for  the  information  obtained  in  the  examples  is  a consultant  interacting 
with  a knowledgeable  person  on  the  problem  characteristics.  The  consultant 
elicits  responses  which  provide  the  information  folded  into  a general  decision 
framework . 

A later  ONR  contract  on  Adaptive  Computer  Aiding  in  Dynamic  Decision  Processes 
takes  the  prior  work  a step  further  by  automating  the  ability  to  track  the 
responses  of  the  decision  maker  in  real  time  in  order  to  learn  the  decision 
strategy,  and  then  to  assist  the  decision  maker  in  optimizing  the  process.  This 
report  contains  a number  of  bibliographic  references  (many  of  them  ONR  reports). 

The  estimation  of  subjective  probabilities,  in  a sense,  is  almost  a separate 
subject,  and  is  a critical  component  of  the  entire  decision  process.  The 
present  Educational  Technology  Assessment  Model  uses  a very  simplified  method 
in  requesting  three  estimates  of  the  variable;  the  most  likely  magnitude  of 
the  variable,  the  most  pessimistic  estimate,  and  the  most  optimistic  estimate. 
The  assumption  is  that  almost  100%  of  probability  density  function  is  contained 
in  the  interval,  and  that  a Beta  distribution  approximating  the  normal,  results. 
Hahn  and  Shapiro  (1967)  indicate  this  distribution  as  applicable  to  variables 
bounded  at  both  ends.  The  PERT  methodology  for  estimating  project  completion 
times  uses  this  distribution  and,  for  the  interested  reader,  is  discussed  in 
Battersby  (1964).  ETAM  provides  similar  caveats  as  are  contained  in  the 
literature,  cautioning  the  use  of  probabilities  derived  in  this  manner.  Several 
methods  for  improving  subjective  estimates  are  discussed  by  Huber  (1974)  and 
are  also  contained  in  the  Handbook  for  Decision  Analysis.  The  scaling  pro- 
cedures and  the  estimation  of  points  on  a continuum  that  bring  about  agreement 
of  the  model  variables  with  reality,  are  crucial  to  the  validity  of  model  re- 
sults. Further  work  is  proposed  in  this  area  and  is  outlined  in  Section  VII 
of  this  report. 

The  economic  analysis  section  of  the  model  is  straightforward  and  is  based 
upon  widely  accepted  techniques  for  handling  investment  type  decisions. 

SECNAV  INSTRUCTION  7000. 14A  served  as  a basis  for  a portion  of  the  output 
specifications  for  the  results  of  the  economic  analysis.  No  further  literature 
references  are  considered  necessary  here,  although  several  are  contained  in  the 
bibliography  for  the  interested  reader. 
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In  conclusion,  this  overview,  while  brief,  has  maintained  relevance  to  the 
key  issues  appropriate  to  the  project.  One  is  to  establish  a basis  of  aware- 
ness of  the  range  of  extant  literature  potentially  relevant  to  the  solution 
of  the  problems  tackled  here.  It  seems  clear  that  no  systematic  attempt  has 
been  previously  published  to  encompass  the  objectives  posed  by  projecting  the 
potential  implications  of  any  kind  of  innovation  in  instructional  technology, 
nor  the  acquisition  of  a data  base  indexed  to  cope  with  the  search  aspects  of 
such  a task.  But  the  reader  is  pointed  to  the  literature  which  may  be  at 
least  tangentially  related  to  this  enterprise,  if  an  inquiry  to  alternatives 
may  be  considered  desirable.  The  bibliography  presented  at  the  end  of  this 
chapter,  while  small,  points  to  extensive  bibliographies  contained  in  the 
references . 
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SECTION  III 

EDUCATIONAL  TECHNOLOGY  ASSESSMENT  MODEL  PROCEDURES 


The  Educational  Technology  Assessment  Model  (ETAM)  is  a series  of  tasks 
supported  by  subtasks  and  detailed  procedures  intended  to  provide  guidance 
to  persons  assessing  the  value  of  a proposed  "educational  innovation." 

The  value  will  be  determined  by  systematically  considering  the  benefits  and 
liabilities  (including  savings  and  costs),  attendant  risks  from  both  internal 
and  environmental  influences,  the  potential  range  of  applicability,  and  its 
worth  in  terms  of  certain  economic  measures.  A major  determinant  of  true 
value  is  related  to  the  range  of  application  identified  for  the  innovation. 
The  model  outlines  a comprehensive  pattern  of  taxonomic  elements  based  upon 
psychological  learning  principles  and  instructional  economics  which  provide 
the  assessor  with  procedural  guidance  in  establishing  the  innovation  range 
of  application.  The  general  model  structure,  the  major  analytic  and  decision 
components,  the  documentation  features,  and  basic  user  guidance  are  outlined 
in  this  introductory  section. 

APPLICATION  AND  COWON  SENSE 


The  reader  may  be  initially  intimidated  by  the  bulk  of  this  manual.  But, 
like  an  unabridged  dictionary,  only  very  limited  sections  may  be  applicable 
to  a given  problem.  One  reason  for  bulk  is  its  ability  to  respond  to  a very 

large  variety  of  problems.  Another  reason  for  its  bulk  is  that  it  is  not 

only  a structured  classification  system,  it  is  also  a condensed  text  de- 
scription of  the  technology  of  education,  training,  and  learning.  The  text 

is  essential  for  clarifying  the  concepts  that  give  useful  meanings  to  the 

terms  used  in  the  classification.  The  text  itself  is  an  instructional 
document.  A technology  which,  in  application,  results  in  the  spending  of 
tens  of  billions  of  dollars  a year  in  this  country,  deserves  careful  ex- 
position, and  perhaps  especially  to  those  getting  the  data  for  making 
decisions  on  the  funding  of  projects  having  such  financial  and  social 
significance. 


The  reader  who  is  a potential  assessor  of  an  innovation  may  furthermore 
be  intimidated  by  the  apparent  length  and  complexity  of  procedures  for 
collecting  and  analysing  data  for  decision  making.  But  there  is  no  com- 
pulsion to  follow  through  all  stages  and  levels  of  data  collection  and 
analysis.  The  following  imaginary  examples  fit  ETAM  structure. 


1 . Teaching  sleeping  students 

Innovator:  My  machine  can  teach  students  while  they  sleep. 

Assessor:  What  skills  can  it  teach? 

Innovator:  Anything  that  can  be  learned  by  hearing  spoken  words. 

Assessor:  How  much  does  it  cost? 
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Innovator: 

Including  R&D  costs,  procurement  and  maintenance,  $20  per 
student  per  night,  assuming  10,000  student  nights  per  year. 

Assessor: 

What's  the  sleep  loss  effect? 

Innovator: 

Eight  hours  sleep  with  learning  becomes  equivalent  to  seven 
hours  sleep  without  it. 

Assessor: 

If  the  student  were  awake,  how  long  would  it  take  him  to  learn 
what  he  is  taught  in  a night's  sleep? 

Innovator: 

Awake,  it  would  take  him  ten  to  fifteen  minutes  to  learn  what 
he  learned  overnight  in  sleep. 

Decision  maker  (who  overheard  the  conversation):  Forget  it! 

2.  Brain-Coupled  Learning:  The  Enqram  Transmitter 


Innovator: 

This  is  an  electronic  device  that  enables  me  to  link,  by  elec- 
trodes, the  brain  activities  of  a skilled  performer  of  a task 
to  the  brain  of  a student.  When  the  skilled  performer  is  doing 
the  job,  the  impulses  transmit  to  the  student  who  actually  learns 
while  the  instructor  performs.  The  student  can  then  immediately 
perform  the  same  thing  at  the  same  skill  level  as  his  instructor. 

Assessor: 

What  tasks  have  you  actually  taught  this  way? 

Innovator: 

Bicycle  riding,  pursuit  tracking  on  an  oscilloscope,  and  celestial 
navigation.  I also  taught  TV  troubleshooting  with  the  Engram 
Transmitter  but  it  took  quite  a few  sample  exercises  to  do  it. 

Assessor: 

Any  difficulties  in  matching  student  and  teacher? 

Innovator: 

Yes,  great  variability  in  both.  Out  of  a hundred  would-be 
teachers  and  a hundred  would-be  students,  only  about  five  pairs 
can  work  out. 

Assessor: 

What's  the  saving  in  student  training  time  over  normal  times  to 
teach  and  learn  the  same  tasks? 

Innovator: 

It  ranges  from  50%  to  90%  savings  in  terms  of  student  practice 
hours.  I can't  explain  yet  why  it  doesn’t  work  with  all  teacher- 
student  pairs.  You  understand,  the  teacher  is  not  teaching,  he 
is  just  doing  his  job  while  his  brain  "teaches." 

Assessor: 

How  many  dollars  and  time  of  R&D  would  be  needed  for  production 
prototypes  to  be  developed,  with  a 90%  level  of  successful 
matching  of  student  and  teacher? 

Innovator: 

I see  about  $50  million  in  R&D  and  ten  years  of  study  and 
development.  We  still  don't  really  understand  the  phenomenon 
at  work  here. 
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Decision  maker:  There  is  half  a million  in  my  budget  for  a high  risk, 

high  payoff  venture,  especially  if  it's  in  an  entirely 
new  area  of  exploration.  Do  you  want  to  take  that  and 
solve  the  problem  of  matching  the  student  and  teacher  at 
levels  that  approximate  our  present  aptitude  requirements 
for  the  same  job?  I can't  use  a rare-bird  phenomenon,  no 
matter  how  good  it  is. 

3.  Training  in  Handgun  Firing 


Innovator:  I have  a device  and  procedure  to  teach  accuracy  in  firing  hand- 

guns. Controlled  experiments  show  my  method  gets  the  same  level 
of  accuracy  with  5,000  rounds  fired  as  your  training  methods  get 
with  7,500  rounds.  I need  no  more  instructors  than  you  do,  and 
no  more  student  time  than  you  use.  Accuracy  is  measured  by  all 
your  present  criteria.  With  7,500  rounds  fired  by  my  method, 
average  accuracy  of  my  trainees  is  30%  superior  to  yours,  plus 
or  minus  10%,  covering  ninety  percent  of  the  cases.  Skill  re- 
tention over  six  months  without  practice  is  better  by  30%. 

Assessor:  Does  your  method  apply  to  other  aiming  operations  such  as  rifles, 

bazookas,  mortars,  automatic  rifles? 

Innovator:  I don't  know  if  it  can  be  adapted  to  them,  but  I don't  think  so. 


Assessor:  What  does  your  training  cost? 

Innovator:  About  25%  of  your  savings  in  the  cost  of  the  ammunition  alone. 

The  increase  in  skill  and  retention  over  your  method  of  training 
is  a bonus. 


Decision  maker:  If  it  does  what  he  says  it  does,  let's  use  it.  We  can  find 

out  later  if  it  can  be  adapted  to  other  weapons. 


If  one  is  familiar  with  the  procedures  and  philosophy  of  ETAM,  there  is 
nothing  inconsistent  between  these  scenarios  and  those  procedures.  The 
innovator  and  the  assessor  quickly  got  to  the  essential  point  of  each 
decision:  no  go;  study  it  further;  or  accept.  In  the  first  example,  it 
was  quickly  evident  that  the  costs  overwhelmed  even  a potential  benefit, 
and  in  any  event  the  benefit  was  trivial. 

In  the  Engram  Transmitter,  the  potential  value  of  the  innovation  could  be 
enormous  as  evidenced  by  the  savings  in  training  time,  but  the  key  risk 
factor  was  in  its  range  of  applicability  to  pairings  of  teacher  and  student. 
No  doubt  additional  studies  on  what  could  and  could  not  be  learned  this  way 
would  also  need  to  be  done  before  further  large  conmitments.  In  the  hand- 
gun trainer,  the  ratio  of  costs  to  benefits  is  clear,  and  the  range  of 
applicability,  although  highly  limited,  is  sufficiently  important  to  justify 
an  acceptance  decision. 
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The  ETAM  procedures  emphasize  that  investigation  can  be  halted  when  infor- 
mation is  adequate  for  the  human  decision  maker  to  accept,  reject,  or 
study  further  the  proposed  innovation.  Extensive  assessment  examination 
is  justified  to  the  degree  that: 

1.  The  economic  and  social  commitment  is  very  large. 

2.  There  is  large  uncertainty  and  risk  in  the  success  of  the  pro- 
posed innovation  as  an  enterprise. 

3.  The  advantage  of  benefits  to  costs  is  marginal  or  dubious, 
or  many  of  the  benefits  are  intangibles. 

4.  The  training  objective  is  extremely  important  and  no  technical 
alternative  seems  to  be  available. 

CHARACTERISTICS  AND  EXAMPLES  OF  TRAINING  INNOVATIONS 

WHAT  IS  AN  INNOVATION  IN  TRAINING  AND  EDUCATION?  An  innovation  is  a relatively 
constant  or  enduring  change  in  the  procedures,  objects,  or  functions  used  in 
any  aspect  of  the  instructional  process,  which  may  be  viewed' as  a benefit  (or 
a liability)  and  has  associated  costs.  Thus,  an  innovation  may  be  a technical 
invention  or  extrapolation  from  an  invention,  or  a structural  modification  to 
the  setting  of  instruction,  such  as  classroom  to  on-the-job. 

The  following  are  general  classes  of  factors  within  and  among  which  innova- 
tions can  be  introduced. 

1.  Change  in  content  of  instruction  for  a given  set  of  terminal 
objectives. 

2.  Change  in  media  or  device  used  as  a vehicle  for  delivering 
instruction. 

3.  Modifications  in  delivering  instructional  or  learning  process 
functi ons. 

4.  Shift  in  the  environment  in  which  a given  phase  of  instruction 
is  given. 

5.  Shift  in  the  characteristics  of  input  students  to  one  or  a set 
of  courses. 

6.  Shift  in  policy  between  student  performance  objectives  to  be 
acquired  at  school  and  those  acquired  on  the  job. 

7.  Change  in  criterion  levels  of  performance  required  to  complete 
a course,  or  to  graduate  from  a school. 

8.  Change  in  technique  for  presentation  of  content  for  more  effi- 
cient learning  and  transfer  of  a given  class  of  material. 

9.  Modification  of  processes  for  generating  training  requirements 
and  objectives  and  for  converting  this  information  into  instruc- 
tional  content  and  test  criteria. 
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10.  Change  in  the  organization  of  Instructional  modules  as  elements 
in  programs  of  instruction. 

11.  Change  in  selecting,  training,  and  assigning  of  instructors. 

Any  list  of  things  about  which  innovation  is  possible  must  be  incomplete. 

The  preceding  list  is  no  exception. 

EXAMPLES  OF  TRAINING  INNOVATIONS.  The  following  are  some  specific  examples 
of  innovations.  It  is  immaterial  that  some  are  hypothetical  proposals. 

Static  Rehearsal  of  Complex  Skills.  Experienced  pilots  had  learned  in  a 
simulator,  space  vehicle  approaches  and  landings  under  instrument  and  visual 
conditions.  The  innovation,  consisting  of  techniques  of  self-imposed  verbal 
rehearsals  of  the  procedures  in  these  maneuvers,  showed  as  much  retraining 
benefit  as  the  control  subjects  who  practiced  the  retraining  in  a simulator. 

Systems  Effectiveness  Analyzer  (SEA) . SEA  utilizes  existing  equipment,  plus 
computer-based  software,  to  enhance  existing  AMRL  capabilities  to  evaluate 
weapon  system  performance.  It  assesses  the  setup  and  other  conditions  of  the 
system  as  the  human  operates  it  procedural ly,  and  also  assesses  the  perfor- 
mance of  the  human  operator  by  round-by-round  probability  of  kill,  and  the 
resulting  target's  probability  of  survival.  Human  performance  is  evaluated 
in  the  context  of  system  performance. 

Scan  and  Detect  Training.  Although  the  radar  screen  is  the  primary  target 
for  this  proposal,  any  task  that  emphasizes  the  scanning  of  a perceptual 
field  and  the  detection  of  a class  of  signal  within  a context  of  clutter  is 
a candidate.  The  sequence  of  training  is  as  follows:  the  student  is  pre- 

sented with  the  signal  to  be  detected  in  a field  without  background  noise; 
immediately  thereafter,  the  same  signal  is  presented  in  a series  of  increas- 
ingly heavy  background  noise  or  disturbance;  the  type  of  noise  is  varied. 

The  student  learns  not  merely  to  detect  the  signal  in  ideal  (and  unrealistic) 
conditions,  but  rather  to  read  through  noise.  The  noise  introduced  becomes 
greater  than  encountered  most  of  the  time,  so  the  student  becomes  "over- 
trained" in  reading  through  noise,  leading  to  a high  level  of  skill  and 
reliable  performance  in  reading  through  "moderate"  noise  levels.  Reading 
through  noise  for  a given  class  of  signal  in  a given  sensory  modality,  is 
presumed  to  be  generalizable  as  a skill.  The  psychological  principle  is  that 
of  learning  quickly  to  suppress  perceptual  ground  irrelevance  as  a "format" 
for  the  perceptual  tasks.  This  is  applicable  to  auditory  as  well  as  visual 
tasks. 

Student  Participation  in  Learning  Route.  When  a student  enters  his  first 
orientation  course  for  training  in  a career  specialty,  he  is  given  a flow 
chart  or  critical  path  structure  (with  alternative  branch  points)  that  will 
lead  him  through  training  to  a full-fledged  competence.  The  student  can  see 
where  he  is  on  this  path,  and  what  lies  ahead.  He  can  interrogate  points 
along  the  path  for  one  or  more  levels  of  conceptual  presentation  of  the  sub- 
ject matter  and  skills  he  is  to  learn  --  and  these  skills  are  shown  manifest 
in  job  situations.  The  student,  therefore,  can  see  the  outcomes  of  his 
learning  efforts  for  what  he  has  yet  to  learn  and  this,  in  turn,  to  its 
meaning  in  job  operations.  This  presentation  of  critical  path  in  training 
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should  also  enable  the  student  to  access  at  any  point,  further  levels  of 
detail  of  job  information  (or  job-training  information),  either  for  review 
or  as  "advance  organizers"  of  future  learning.  The  outcome  should,  for 
most  students,  be  increased  motivation  to  learn  as  manifest  in  (a)  more 
rapid  learning  of  course  content;  (b)  better  retention  of  course  content  over 
periods  of  time;  (c)  better  transfer  of  material  learned  in  previous  courses 
to  new  courses;  and  (d)  higher  performance  levels  on  performance  tests. 

The  psychological  principle  used  as  the  rationale  here  is  that  the  student 
is  an  active  participant  in  his  learning  process,  has  greater  mnemonic 
richness  while  learning  (with  effects  in  learning  rate,  retention,  and 
recall),  and,  ultimately,  more  motivation  on  the  job. 

Predicting  Student  Performance  Levels  from  Types  of  Learning  Errors.  A 
technique  is  devised  for  capturing  and  using  the  kinds  of  errors  (as  well  as 
their  relative  frequencies)  made  by  students  in  the  process  of  learning. 

Error  measurement  is  begun  on  their  first  day  in  any  class,  and  is  cumulative 
in  building  a profile.  Some  kinds  of  error  are  highly  predictive  of  success 
in  a given  kind  of  duty,  depending  on  where  in  the  course  of  learning  they 
occur.  Others  are  predictive  of  failure  in  a given  kind  of  performance 
expectation,  but  may  predict  success  in  other  tasks  or  performance  expecta- 
tions. Still  other  kinds  of  errors  have  no  predictive  value.  Error  data  are 
combined  with  learning  data. 

The  error  analysis  technique,  therefore,  predicts  washouts,  and  more  or  less 
precisely  where  the  washout  will  occur  in  training  (or  on  the  job).  Thus,  it 
can  predict  the  student's  ceiling  in  a class  of  activity.  The  error  data  also 
predict  the  kinds  of  tasks  which  the  student  may  be  expected  to  learn  to  do 
well  — thus,  it  is  a selection  procedure  and  may  result  in  more  precise 
student  guidance.  It  should  be  emphasized  that  the  appearance  of  large 
amounts  of  given  error  data  early  in  learning  a given  task,  may  be  predictive 
of  more  eventual  success  on  that  task  than  fewer  errors  of  that  kind. 

Error  analysis  depends,  in  part,  on  strategy  in  designing  tests  and  learning 
situations  for  the  test.  It  does  not  add  to  overall  training  time  in  the 
traditional  form  of  instruction.  It  does  add  perhaps  to  the  cost  of 
preparing  training  materials  in  terms  of  specialized  instructor  time.  It 
also  adds  costs  to  the  assessment  process  during  training. 

The  technique  is  derived  from  learning  theory  and  lab  data  that  show  different 
aptitudes  coming  into  dominance  at  different  stages  of  learning.  But  aptitudes 
interact  with  task  content  and  with  method  of  training  and  learning. 

Key  Personnel  Identification  and  Special  Training.  Considerable  attention 
and  resource  Is  given  in  formal  education  to  the  slow  learner.  But  some 
individuals  entering  training  are  far  more  likely  than  others  to  become 
"key  persons"  within  (or  even  outside)  their  trained  skill  specialties  in 
their  operating  organizations,  and  training  adapted  to  their  discovery  and 
early  cultivation  can  have  high  payoff.  Factors  in  addition  to,  or  other 
than,  ordinary  technical  skills  lead  to  key  roles.  Early  training  may 
include  identifiers  for  these  student  properties,  and  lead  these  students 
through  special  pathways  that  enhance  their  ability  to  assume  key  roles 
quickly  when  on  the  job,  rather  than  depend  on  extensive  proving  and 
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on-the-job  training  experiences.  The  key  person  may  actually  show  up  more 
poorly  in  the  learning  of  some  classroom  materials  and  sub tasks  than  many 
individuals  who  turn  out  mediocre  on  the  job. 

The  learning  strategy  is  not  fast-track,  but  a different  track  and  style 
of  instruction.  It  would  need  to  be  individualized  or  taught  through  small 
group  coaching;  have  minimized  criteria  on  the  "mechanics"  content  of  train- 
ing; and  emphasize  individual  initiative  in  learning  what  is  situationally 
needed;  and  in  planning,  organizing,  and  evaluating  objectives. 

The  training  costs  for  these  students  will  be  high,  and  the  benefits  will 
not  show  up  in  standard  performance  tests,  but  only  in  job  behaviors  in  the 
field.  The  washout  and  dropout  rate  may  be  high.  It  is  estimated  that 
about  5%  of  an  ordinary  class  of  trainees  have  key  person  potential,  and  out 
of  every  10  key  persons  selected,  only  8 will  achieve  key  competence. 

General  Comment.  The  structuring  of  a technology  that  both  articulates  and 
integrates  functional  entities  may,  in  itself,  be  a source  of  invention 
and  innovation.  Even  though  ideas  may  be  serendipitous,  they  may  be  developed 
and  tested  systematically.  It  also  becomes  feasible  to  fit  research  findings 
into  application  frameworks,  and  to  extrapolate  the  significance  of  an  engineer- 
ing development,  such  as  low-cost  microelectronics  and  display  technologies, 
to  instructional  functions  long  before  specific  applications  are  designed  with 
them.  Thus,  an  accelerating  tempo  of  cost-effective  innovation  can  be  achieved. 

For  example,  the  adaptation  of  computer  technologies  to  education  and  training 
is  a costly  undertaking  and  can  generally  be  cost  justified  only  when  there  is 
sufficient  scale  of  utilization.  The  application  must  achieve  a "critical 
mass"  of  usefulness  to  a wide-range  of  students  with  a wide-range  of  practical 
task-learning,  through  a wide-range  of  levels  of  skill  acquisition,  before  the 
payoffs  can  mount  up  dramatically.  This  is  one  example  of  the  need  for 
systematization  of  a technology  in  order  to  optimize  the  full  potential  range 
of  resources  within  that  technology  for  coping  with  the  objectives  of  the 
large  enterprise. 

MODEL  STRUCTURE 

The  model  consists  of  eight  distinct  tasks  to  guide  the  assessor  in  creating 
a comprehensive  description  of  the  proposed  educational  innovation,  in  iden- 
tifying potentially  more  cost-effective  alternatives,  in  determining  the  full 
range  of  application  for  the  innovation,  in  assessing  the  risks,  benefits, 
and  liabilities  associated  with  the  implementation  and  use  of  the  innovation, 
and  in  preparing  a coherent  presentation  of  the  analytic  results  for  review 
by  the  ultimate  decision  maker.  The  individual  tasks  and  their  interrela- 
tionships are  shown  in  Figure  1 1 1 - 1 . The  tasks  are  designed  and  sequenced 
to  accomplish  two  primary  objectives. 

• First  is  to  provide  a means  to  determine  the  true  value  of  the 
innovation  if  it  is  applied  to  the  training  situation. 

• Second  is  to  find  the  full  range  of  potential  applications  for 
the  innovation  in  the  training  environment. 
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FIGURE  111-1.  ETAM  TASK  FLOW 
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In  meeting  these  two  objectives,  the  ETAM  procedures  can  be  envisioned  as 
having  two  major  components.  The  first  consists  of  those  activities  which 
permit  expansion  of  the  innovators  view  as  to  how  and  where  the  innovation 
might  be  applied.  This  type  of  "divergent  activity"  takes  place  at  the 
beginning  of  the  procedural  process  by  means  of  the  expanded  description 
of  the  innovation  and  through  the  range-of-effect  analysis.  Once  the  full 
potential  for  applying  the  innovation  has  been  explored,  the  second,  or 
"convergent  activity,"  deals  with  assessing  all  aspects  of  the  innovation 
across  this  full  range  of  application.  The  second  objective  is  to  converge 
to  the  decision  to  accept  or  reject  the  innovation.  This  conceptual  view 
of  the  ETAM  procedures  is  shown  in  Figure  IJI-2. 

Each  of  the  tasks  comprising  the  Educational  Technology  Assessment  Model  is 
briefly  described  below: 

9 Task  1 - Formalize  the  Description  of  the  Innovation.  The  in- 
novator is  provided  with  a number  of  categories  for 
systematically  expanding  the  descriptive  information 
on  the  innovation.  The  objective  of  the  innovation, 
target  applications,  and  the  results  of  empirical 
studies  involving  limited  application  of  the  innovation 
are  among  the  categories  to  be  completed. 

• Task  2 - Develop/Examine  Alternatives  to  the  Innovation.  In 

this  task  the  innovator  is  requested  to  consider  pos- 
sible alternatives  to  the  proposed  innovation  which 
may  require  a lesser  level  of  investment  funding,  and 
possibly  be  more  cost-effective. 

• Task  3 - Make  Preliminary  Feasibility  Profile.  The  formal 

assessment  procedure  begins  at  this  point  with  a con- 
sideration of  potential  risks  in  the  implementation 
and  application  of  the  innovation,  due  to  such  factors 
as  organizational  incompatibilities,  goals/policies 
incompatibilities,  technological  requirements,  technical 
support  requirements,  funding  constraints,  and  problems 
in  attitudinal  acceptance  by  users. 

• Task  4 - Perform  Analytic  Feasibility  Assessment.  The  results 

of  Task  3 are  analyzed  in  greater  depth,  and  risk  re- 
duction studies  and  projects  are  defined.  Risk  assess- 
ments are  made  in  view  of  these  studies  and  projects, 
and  a preliminary  decision  tree  is  structured  for  pro- 
viding initial  guidance  as  to  whether  the  innovation 
should  be  accepted  outright,  accepted  with  the  addi- 
tional expenditures  for  the  risk  reduction  projects, 
or  rejected. 

• Task  5 - Determine  Range-of-Effect.  The  prior  tasks  have  been 

concerned  with  the  assessment  of  the  innovation  over  a 
limited  range  of  application.  Initially,  it  was  with 
the  target  applications  identified  by  the  innovator; 
in  Task  4 a preliminary  extended  range  of  application 
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was  considered.  Task  5 contains  a number  of  classifi- 
cation structures  to  which  the  innovation  is  related. 

Each  structure  may  result  in  the  assignment  of  one  or 
more  descriptors  applicable  to  the  innovation.  These 
descriptors  will  serve  as  search  parameters  in  finding 
the  full  range  of  courses,  jobs,  or  equipments  to  which 
the  innovation  may  potentially  be  applicable.  Task  5 
also  deals  with  describing  the  benefit  pattern  which 
may  be  expected  of  the  innovation  when  generally  applied. 

• Task  6 - Perform  Cost-Benefits  Analysis.  The  decision  tree  de- 

veloped in  Task  4 is  refined  to  include  more  precise  costs 
and  savings  derived  from  processing  the  tangible  bene- 
fits through  cost  models.  In  addition,  the  intangible 
benefits  are  so  analyzed  that  they  are  expressed  in 
"equivalent  dollars,"  so  that  the  various  decision  paths 
can  be  more  easily  evaluated.  Equivalent  dollars  is  a 
utility  expression  rather  than  a literal  dollar  value. 
Probabilities  of  implementation  success  and  user  accep- 
tance with  and  without  the  risk  reduction  projects  are 
refined,  and  the  decision  variables  ar-e  recalculated. 

• Task  7 - Perform  Financial  Analysis.  This  task  is  concerned 

with  assessing  the  tangible  benefits  and  liabilities 
(those  expressable  in  real  dollars)  in  terms  of  certain 
economic  measures.  The  investment  costs  and  the  annual 
costs  and  savings  are  calculated  over  a planning  period 
extending  a number  of  years  into  the  future.  Rates-of- 
return  on  the  invested  dollars  are  determined  for  the 
Incremental  effect  of  each  alternative  compared  to 
every  other  alternative.  Alternatives  consist  of  the 
proposed  innovation,  the  existing  system,  and  any 
other  approaches  defined  In  Task  2 which  were  con- 
sidered reasonable  candidates  for  further  assessment. 

This  assessment  process  provides  a separate,  distinct 
view  of  the  value  of  the  innovation  as  was  gained  from 
the  decision  tree  assessment  in  Task  6.  Both  are  in- 
puts to  the  decision  maker. 

• Task  8 - Make  Accept/Reject/Study  Decision.  The  decision  informa- 

tion and  formats  have  been  addressed  throughout  the 
assessment  process.  This  task  provides  guidance  to  the 
presenter  of  the  analytic  data  to  the  ultimate  decision 
maker. 

The  model  is  structured  so  that  the  outputs  of  each  task  will  permit  a deci- 
sion as  to  whether  the  assessment  process  should  be  continued.  The  level  of 
effort  associated  with  the  carrying  out  of  any  task  should  be  consistent 
with  the  potential  value  of  the  innovation  indicated  at  that  point. 

ANALYTIC  AND  DECISION  COMPONENTS 

The  model  design  provides  a hierarchical  analytic  approach  and  a sequential 
decision  structure.  The  analysis  proceeds  from  judgmental  inspection  of 
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descriptive  data  to  the  application  of  more  sophisticated  quantitative 
techniques  intended  to  provide  the  decision  maker  with  a more  extensive 
understanding  of  the  innovation's  potential  value.  Each  analytic  step  in 
the  assessment  process  expands  the  knowledge  base  for  decision  making,  and 
should  provide  a higher  level  of  confidence  in  any  decision  indicated.  At 
the  same  time,  the  process  provides  an  Increasing  awareness  of  the  uncer- 
tainties inherent  in  Information  to  be  presented  to  the  decision  maker. 

Both  the  expanded  knowledge  base  and  the  increased  awareness  of  uncertainty 
become  essential  components  of  the  Information  provided  the  decision  maker. 
There  is  no  requirement,  however,  that  all  of  the  tasks  be  carried  out  be- 
fore sufficient  information  is  available  for  a decision.  The  .tasks  are 
sequenced  so  that  an  accept/reject  decision  can  potentially  be  made  at  any 
point  in  the  procedure. 

ANALYTIC  FEATURES.  The  initial  analysis  centers  around  the  formalized  de- 
scription of  the  Innovation.  The  objective,  assumption,  target  applica- 
tions, and  empirical  data  can  be  examined  to  determine  whether  further 
efforts  should  be  applied  to  the  assessment  of  the  proposed  innovation. 
Obvious  risks  in  the  implementation  and  use  of  the  innovation  which  might 
prevent  intrinsic  benefits  from  being  realized  are  next  considered  in  the 
analytic  process.  The  opinions  of  experts  are  sought  in  obtaining 
estimates  of  success  probabilities  for  implementation  and  user  acceptance. 

A decision  tree  provides  this  framework. 

If  unusually  high  levels  of  risk  result  from  this  preliminary  feasibility 
profile,  possible  approaches  for  reducing  the  level  of  risk  are  then 
identified.  These  will  introduce  additional  costs  which  must  be  con- 
sidered in  the  overall  cost  analysis.  At  the  same  time,  an  expanded  range 
of  application  for  the  innovation  is  considered  in  assigning  costs  and 
savings  to  the  decision  tree.  The  next  step  In  the  analysis  Is  to  deter- 
mine more  precisely  where  the  proposed  innovation  might  be  applied. 
Taxonomic  structures  are  used  to  classify  the  innovation  so  that  a search 
of  various  data  bases  containing  target  applications  indexed  with  the  same 
taxonomic  descriptors,  will  result  In  identification  of  a potentially  wide 
range  of  uses  for  the  innovation.  This  analytic  process  provides  a 
systematic  technique  for  generalizing  the  use  of  the  innovation  in  order 
to  complete  the  assessment  process  in  Its  full  potential.  A detailed  cost 
analysis  is  now  conducted  for  the  full  range  of  courses,  jobs,  equipments, 
etc.,  to  which  the  innovation  might  be  applied.  Important  benefits  and 
liabilities  are  Introduced  into  this  phase  of  the  analysis  by  assigning 
them  equivalent  dollar  values.  The  benefits  and  liabilities  quantified 
in  this  manner  are  combined  with  the  actual  savings  and  costs.  Equivalent 
dollar  values  are  identified  for  each  possible  outcome  on  the  decision 
tree  so  that  the  analytic  framework  permits  calculation  of  decision  vari- 
ables which  express  a more  comprehensive  measure  of  the  utility  or  value 
of  a particular  decision.  This  level  of  analysis  is  complemented  with  an 
financial  analysis  of  the  proposed  innovation  and  possible  alternatives. 
Measures  of  financial  value  are  developed  as  a part  of  this  analytic 
process.  The  full  range  of  analysis  discussed  above  has  a number  of 
decision  points  Interposed  in  order  that  subsequent  analysis  at  more  in- 
depth  levels  will  be  justified  by  any  decision  to  proceed  beyond  that  point. 

DECISION  FEATURES.  From  the  preceding  discussion,  the  sequential  nature  of 
the  decision  structure  should  be  more  or  less  obvious. 
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The  expanded  description  of  the  innovation  assists  in  deciding  to  continue 
or  abandon  the  assessment  process  at  that  point.  The  risk  analysis 
identifies  any  serious  obstacles  to  the  successful  implementation  and  use 
of  the  innovation;  this  occurs  early  in  the  assessment  process,  prior  to 
the  gathering  of  extensive  range-of-effect  and  cost  data.  The  decision 
tree  framework  forces  the  assessor  to  consider  the  implications  of  a number 
of  possible  outcomes  in  the  decision  process.  For  example,  the  effects  of 
successfully  implementing  the  innovation  only  to  have  it  fail  to  gain 
adequate  user  acceptance  may  be  so  serious  (excessive  recovery  costs,  in- 
adequately trained  personnel,  long-term  potential  for  rejecting  subsequent 
innovations  which  might  have  otherwise  been  successful,  etc.)  that,  con- 
sidering the  success  probabilities,  the  decision  to  accept  may  be  judged 
too  risky  at  this  point  in  the  evaluation.  Identification  of  the  full  range 
of  application  for  the  innovation  permits  decisions  and  planning  efforts  to 
be  performed  more  realistically  at  the  beginning,  even  though  a resulting 
decision  may  be  to  implement  in  phases.  Two  independent  views  of  the 
innovation's  value  are  made  available  to  the  decision  maker  in  the  decision 
tree  framework  and  in  the  economic  analysis.  The  decision  tree  is  refined 
to  assist  the  decision  maker  to  select  the  path  with  the  highest  payoff, 
taking  into  account  both  tangible  and  intangible  benefits  and  liabilities. 
The  decision  path  resulting  from  this  stage  of  analysis  becomes  the  basis 
for  the  comparative  analysis  of  the  proposed  innovation  against  the 
existing  system,  and  also  against  any  reasonable  alternatives.  The  eco- 
nomic analysis  satisfies  the  requirement  of  the  financial  component  in  the 
decision  process  by  ensuring  that  potential  alternatives  for  investment  of 
scarce  resources  have  been  fully  explored,  and  each  has  been  subjected  to 
evaluation  to  determine  the  respective  returns  on  invested  capital.  The 
final  decision  documentation  is  organized  so  that  the  decision  maker 
obtains  a clear  picture  of  the  merits  and  shortcomings  of  the  innovation, 
and  also  a sound  understanding  of  the  validity  of  the  data  used  in  the 
assessment  and  of  the  degree  of  uncertainty  associated  with  the  total  range 
of  decision  information. 

MODEL  DOCUMENTATION 

The  entire  procedural  model  for  assessment  of  educational  technology  has 
been  designed  with  several  unique  documentation  features  to  enhance  both 
its  usage  and  its  maintenance. 

TASK  PROCEDURAL  HIERARCHY.  An  overall  task  flow  of  the  model  has  been  pre- 
sented in  this  section  of  the  model  documentation.  This  is  intended  for 
those  interested  only  in  an  overview  of  the  model  structure.  Each  task,  In 
turn,  has  been  documented  at  several  levels  of  detail.  The  top  level 
consists  of  flows  showing  the  subtasks  to  be  undertaken  by  the  assessor  at 
that  phase  of  the  assessment.  The  sequential  or  parallel  accomplishment 
of  the  subtasks  can  be  determined  by  referring  to  this  level  of  documenta- 
tion. Also,  the  task  flows  show  decision  points  internal  to  the  task 
which  direct  the  assessor  to  alternate  subtasks.  Following  the  task  flow 
is  an  overall  task  description  containing  a list  of  the  subtasks  within 
that  task  category.  Task,  sub task,  and  page  references  are  provided  to 
permit  rapid  access  to  procedural  information.  This  referencing  scheme 
also  permits  pointing  the  assessor  to  a procedure  contained  within  another 
task,  thus  eliminating  the  need  for  redocumenting  common  procedures.  The 
most  detailed  level  of  documentation  is  the  actual  procedure  for  accomplish- 
ing the  subtask.  A description  section  is  optional  in  Introducing  the  sub- 
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task  procedure  which  will  contain  one  or  more  steps  outlining  how  the  sub- 
task should  be  accomplished.  Comments  are  inserted  within  the  procedural 
steps  when  it  is  felt  a clarification  or  further  reference  is  necessary. 

An  optional  section  is  available  for  a discussion  of  the  rationale  for 
undertaking  the  subtask.  This  will  be  used  when  the  reason  for  the  pro- 
cedure may  not  be  apparent  to  the  user. 

NUMBERING  OF  TASKS  AND  SUBTASKS.  ETAM  consists  of  eight  distinct  tasks 
numbered  1 through  8.  Within  a task,  the  sub tasks  are  numbered  sequentially. 
The  reference  to  Task  5,  Subtask  11,  would  be  05.11.  Each  page  within  a 
subtask  is  numbered  sequentially,  so  the  reference  to  Task  6,  Subtask  2, 
page  3,  would  be  06.02.03.  The  introductory  flows  and  task  descriptions 
containing  a list  of  the  subtasks,  are  identified  as  Subtask  0,  therefore, 
the  reference  to  the  overall  flow  for  Task  4 would  be  04.00. 

The  overall  flows  are  not  given  a figure  number;  however,  an  individual 
figure  within  the  procedural  sequences  is  numbered  to  correspond  with  the 
numbering  of  the  subtask  in  which  it  appears.  Therefore,  the  third 
figure  to  appear  within  Task  6,  Subtask  7,  would  be  labeled  Figure  6.7.3. 

All  figures  within  a subtask  procedure  are  placed  after  the  procedural  steps. 
Certain  figures  may  be  duplicated  without  the  figure  number  designation. 

This  is  to  permit  the  figure  to  be  copied  for  use  by  the  assessor  in  docu- 
menting some  aspect  of  the  assessment  process. 

It  must  be  underscored,  ho wever,  that  the  task  Hows  are  generalized  con- 
ceptual sequences.  Practical  factors  can  justify  departures  from  the 
sequence  expressed  in  the  conceptual  model  when  actual  assessment  procedures 
are  performed.  The  conceptual  map  should  not  be  treated  as  a procedural 
straightjacket. 

The  documentation  design  is  intended  to  allow  ease  in  updating  and  maintain- 
ing the  document.  Changes  can  be  made  with  minimal  impact  upon  other 
sections.  As  additional  information  is  obtained  through  the  use  of  the 
model,  it  can  be  inserted  at  the  appropriate  point. 

USER  GUIDANCE 

The  use  of  ETAM  procedures  requires  an  expert  of  the  consul tant  type  to  re- 
la  te-l!ie-formirri zeTTescrTptToir^f'l.he  innovation  — and  any  data  available 
about  it  --  to  the  full  range  of  instructional  devices,  courses,  and  course 
objectives,  jobs  and  job  tasks,  and  student  characteristics.  The  classifica- 
tion system  that  guides  the  assessor  is  a comprehensive,  procedurally- 
oriented  taxonomic  structure  of  processes  and  products  in  the  economics  of 
instruction  and  in  the  psychology  of  learning.  The  analytic  techniques  re- 
quire a degree  of  understanding  of  decision  theory,  statistical  techniques, 
and  financial  and  economic  analysis"  In  general, therefore,  the  use  of  ETAM 
is  not  intended  for  either  the  trainer  or  the  training  officer.  However,  a^ 
sound  understanding  of  the  ETAM  process  by  training  personnel,  interfacing 
with  an  assessor-consultant,  in  evaluating  an  innovation  is  essential" 

The  ETAM  procedures  have  been  documented  at  several  levels  of  detail,  as 
previously  discussed,  so  that,  initially,  the  assessor  will  have  available 
step-by-step  procedures  for  accomplishing  the  assessment.  As  competency  is 
gained  in  the  use  of  the  procedures,  succeeding  assessment  exercises  may  re- 
quire reference  only  to  the  higher  level  task  descriptions  or  task  flows. 
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The  tasks  within  ETAM  suggest  a form  of  project  management,  including  the 
creation  and  maintenance  of  a project  file.  As  each  task  Is  completed,  its 
outputs  should  be  retained  in  the  project  file  since,  in  many  cases,  sub- 
sequent tasks  build  upon  or  refine  the  results  of  prior  tasks. 

While  systematic  procedures  and  formal  quantification  techniques  may  seemingly 
enhance  the  credibility  of  outputs,  the  innovator,  the  assessor,  reviewers, 
and  decision  makers  are  cautioned  here  that  the  analytic  results  are  noTetter 
than  data  used  in  producing  the  results.  In  fact,  tne  systematic  processing 
of  bad  data  may  produce  outputs  which  are  totally  incongruous  with  sound 
Intuitive  judgment.  All  data  estimates  should  be  continually  subjected  to 
tests  of  reasonableness,  as  should  Interim  model  outputs.  Data  sources 
should  be  documented  in  all  cases  and  verified  for  reliability. 

The  final  note  to  the  user  concerns  the  general  application  of  the  procedures 
to  the  assessment  process.  Each  task,  subtask,  and  procedure  represents  only 
an  approximate  methodology  for  conducting  the  evaluation.  Different  innova- 
tions will  require  variations  in  the  application  of  certain  procedures. 
Additional  procedural  techniques  may  have  to  be  created  by  tne  assessor. 

For  example,  a general  decision  tree  model  has  been  suggeste4  for  use;  how- 
ever,  as  is  pointed  out  within  the  text  of  the  model,  other  decision  tree 
structures  may  need  to  be  created  to  handle  some  types  of  assessment  problems. 
Also,  the  assessor  has  a wide  latitude  as  to  the  depth  to  which  any  particular 
assessment  step  is  pursued"  This  will  be  a function  of  need,  interest,  poten- 
tial value,  risk,  and  the  time  and  resources  available  for  assessment.  At 
each  step  In  the  procedure  the  assessor  should  ask  the  following  questions: 

1)  Do  I have  sufficient  Information  to  reasonably  recommend  rejection  of  the 
Innovation  and  halting  of  further  analysis,  and  2)  can  this  particular  step 
be  simplified  or  bypassed,  or  Is  a more  in-depth  analysis  than  Is  outlined 
suggested? 

A last  explicit  caveat.  An  assessment  team  with  a broad  technical  knowledge 
base  and  penetrating  insights  can  use  ETAM  as  a tool,  develop  a wider  base 
of  alternatives  to  the  innovation,  and  an  awareness  of  range  of  benefits  and 
liabilities  than  will  an  assessment  team  which  lacks  these  capacities  for 
creative  insight.  There  is  no  guarantee,  for  example,  that  a given  training 
aid  will  also  be  perceived  by  an  assessor  as  a job  aid  having  a performance 
benefit,  as  well  as  a training  benefit.  ETAM  is  a structure  for  guiding 
the  collection  and  organization  of  descriptive  and  evaluative  Information  by 
the  assessment  team,  but  it  cannot  generate  that  information.  The  ETAM 
category  structure  does,  however,  guarantee  at  least  a lowest  common  denomi- 
nator of  inquiry  and  data  adequacy  for  assessment  purposes  without  inhibiting 
higher  levels  of  competence. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 

The  description  achieved  by  means  of  this  step  will  lead  to  the  selection  of 
the  appropriate  decision  format  for  determining  the  ultimate  "range-of-effect" 
or  relevance  of  the  innovation.  The  description  is  intended  to  get  as  much 
information  about  the  intent  and  application  of  the  innovation,  as  well  as 
background  data  and  empirical  data,  direct  and  Indirect,  that  the  innovator 
has  developed.  Wherever  the  following  format  is  incomplete,  the  innovator 
should  improvise  or  add  commentary. 

One  justification  in  attempting,  in  general,  to  follow  the  classification 
outline  is  to  simplify  entering  and  retrieving  the  innovation  into  a data 
bank  for  cross-referencing  purposes. 

Descriptions  of  what  is  known  about  each  of  the  following  items  should  be 
entered  under  the  designated  headings  that  are  applicable. 

SUBTASK 

1.  State  the  objectives  and  a summary  of  the  innovation. 

2.  Describe  delivery  system  or  training  vehicle. 

3.  Describe  jobs  and  tasks  applicable  to  the  innovation. 

4.  Describe  relevant  student  attributes. 

5.  Describe  empirical  data  on  benefits  and  savings. 

6.  Estimate  the  practical  importance  of  the  training 
product  resulting  from  the  innovation. 

7.  Estimate  resource  requirements  needed  for  further  study 
and  implementation  of  the  innovation. 

RATIONALE 

All  of  the  information  available  to  and  used  by  the  innovator  should  be 
captured  and  set  down.  The  structure  offered  here  will  guide  the  assessment 
process,  even  though  some  of  the  information  offered  in  this  section  may  have 
to  be  modified  in  the  course  of  the  assessment  process.  The  format  for  ob- 
taining and  setting  down  information  in  Task  1 is  a highly  condensed  version 
of  the  complete  procedural  format  for  obtaining  assessment  data.  Even  though 
the  innovation  may,  for  various  reasons,  be  rejected  early  in  the  assessment 
process  for  adoption,  the  complete  record  of  the  innovator  as  represented  in 
this  task  can  be  retained  in  archives  suitably  indexed  for  specialized  search 
and  retrieval  operations  in  the  event  that  future  developments  warrant  recon- 
sideration, either  on  the  basis  of  other  technical  developments  or  on  the 
basis  of  a changing  need. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  STATE  THE  OBJECTIVES  AND  A SUMMARY  OF  THE  INNOVATION 


PROCEDURE 

1.  The  formal  description  of  the  Innovation  should  be  preceded  by  a brief 
statement  on  its  major  objectives.  This  should  be  followed  by  a one  or 
two  page  suumary  or  abstract  that  describes  the  Innovation  in  general 
terms  and  its  intended  effect(s).  The  sunmary  should  be  completed  with 
a list  of  keyword  descriptors  for  describing  the  Innovation  as  a re- 
trievable item  by  subject  content  in  a data  base  directory. 

COMMENT:  This  set  of  descriptors  may  be  modified  during  and  after  analysis 
of  the  innovation's  range-of-effect. 


III-19 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
01  02  01 


TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  DESCRIBE  THE  DELIVERY  SYSTEM  OR  TRAINING  VEHICLE 


DESCRIPTION 

The  innovation  may  be  directed  towards  the  functional  improvement  or  cost 
reduction  of  methods  of  instructional  delivery  or  of  instructional  media. 

Or  it  may  deal  with  the  process  of  developing  course  materials  or  the 
instructional  process,  either  generally  or  specifically,  to  a course  or  type 
of  course.  Whether  the  effects  may  be  direct  or  indirect,  any  information 
relevant  to  these  factors  known  to  the  Innovator  should  be  identified  here. 

PROCEDURE 

1.  Describe  what  information  is  relevant  to  any  one  or  more  of  the  follow- 
ing topics. 

• Program  or  Course  Design.  Does  the  Innovation  deal  with  the  process 
of  developing  individual  courses,  or  for  a program  of  courses  leading 
to  the  knowledge  and  skill  of  a duty  or  rating,  or  the  design  of  an 
entire  curriculum  structure?  The  emphasis  here  is  technique  for  the 
development  and  design  of  training  or  for  the  organization  and 
modularity  of  training  content. 

• Instructional  Process.  Is  the  innovation  aimed  at  instructional 
technique,  either  through  the  design  of  improved  or  added  functions 
in  the  instructional  vehicle,  or  the  design  and  presentation  of 
content?  Is  a means  made  available  for  teaching  new  levels  or  kinds 
of  skills  or  accelerating  the  rate  at  which  knowledge  or  skill  is 
acquired  or  transferred  to  real -work  environments?  Does  the  innova- 
tion enable  different  locales  for  instruction  to  be  used,  or  spe- 
cialization of  the  content  of  the  instruction  to  the  individual 
student  and  his  particular  needs  and  capabilities?  (This  function 
will  be  repeated  in  the  item  below.)  This  list  is  not  exhaustive. 
Where  an  item  is  applicable,  identify  it  by  name  and  describe  the 
relevant  characteristics  of  the  innovation. 

• Target  Courses.  Was  the  innovation  intended  to  apply  principally  to 
one  course,  or  related  type  of  course?  Identify  the  course  and  the 
specific  course  content  including,  where  possible,  the  name  of  the 
knowledge  or  skill  to  which  it  applies. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  DESCRIBE  JOBS  AND  TASKS  APPLICABLE  TO  THE  INNOVATION 


DESCRIPTION 

In  this  section,  try  to  identify:  (1)  the  specific  job  or  task  in  the  opera- 

tional environment  for  which  the  innovation  is  intended;  and  (2)  by  implica- 
tion, the  range  of  jobs  and  tasks  to  which  the  innovation  applies. 

PROCEDURE 

1.  Use  the  following  terms  and  definitions  to  help  identify  the  jobs  and 

tasks  to  which  the  innovation  applies. 

• Equipment  Orientation.  Does  the  Innovation  apply  to  training  in  the 
operation  or  servicing  of  equipment?  If  so,  is  it  a specific  item 
of  equipment,  or  a class  of  equipment?  Identify  all  particulars. 

t Work  Function  Dealing  with  Equipment.  Does  the  innovation  deal  with 
the  operation  or  the  maintenance  or  service  and  support  tasks  on  the 
equipment  identified  in  the  preceding  item?  In  whatever  case,  specify 
the  applicable  tasks  within  the  Identified  work  function.  Where 
feasible  use  the  Navy's  Manual  of  Qualifications  for  Advancement  as 
a guide  for  identification  and  terminology  of  specific  knowledge  or 
practical  factors. 

• Operational  Environments.  External  operational  environments  may  be 
shipboard,  dock,  bad  weather,  landing  operations,  etc.  Internal 
environments  may  be  described  by  stress,  fatigue,  pain,  or  other 
distress.  An  innovation  may  be  directed  towards  a class  of  external 
or  internal  environment. 

• Target  Job  (Rating,  Billet,  NEC,  NOBC,  etc.).  Where  possible,  specify 
the  applicable  jobs  targeted  by  the  innovation  in  terms  of  Navy  nomen- 
clature; i.e.,  general  rates,  ratings,  etc. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  DESCRIBE  RELEVANT  STUDENT  ATTRIBUTES 


DESCRIPTION 

The  innovation  may  be  directed  primarily  to  the  enhancement  of  desirable 
student  attributes  or  Individual  differences,  or  to  the  minimizing  of 
liabilities.  These  should  be  identified  in  this  subtask. 

PROCEDURE 

1.  Use  the  following  terms  and  definitions  to  help  identify  the  student 
attributes  that  may  be  relevant  to  the  innovation.  Do  not  feel  re- 
stricted to  these  terms  or  concepts  if  others  serve  to  clarify  the  idea. 

• Prior  Training  or  Experience.  Does  the  innovation  enhance,  in  train- 
ing or  performance,  previous  training  and  experiences  or,  conversely, 
is  it  Intended  to  compensate  for  deficiencies  in  previous  training 
and  experience?  These  may  be  done  by  improved  assessment  procedures 
or  training  techniques  or  a combination  of  both.  The  training  may  be 
accomplished  so  as  to  facilitate  Its  transfer,  plus  job  experience, 
to  new  or  extended  training. 

• Attitudes  and  Motivations.  The  Innovation  may  be  directed  towards 
increasing  the  student's  will  to  learn,  interest,  attentiveness,  and 
participation  in  the  learning  process.  These  factors  should  be 
differentiated  from  those  that  may  be  successful  in  recruiting  candi- 
dates for  a given  course  of  instruction.  It  should  be  understood 
that  the  normal  by-products  of  effective  instruction  are  positive 
student  attitudes  and  motivations.  An  innovation  may  be  conceived, 
however,  that  seeks  motivational  enhancement  by  specific  planned 
actions  for  that  particular  purpose,  although  they  may  be  eihbedded 

in  normal  instructional  actions. 

• Aptitudes.  The  target  of  the  innovation  may  primarily  be  that  of 
changing  aptitude  levels  or  mixes  while  holding  the  objectives  of 
training  and  training  costs  constant.  Such  targets  can  be  identi- 
fied, named,  and  usually  quantified. 

• Social  Factors.  Social  environments  of  some  students  may  create 
impediments  to  the  learning  process  and  the  learned  product.  They 
may  also  affect  the  social  climate  in  the  training  environment  and 
later  the  operational  environment.  An  innovation  may  be  aimed  at 
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overcoming  general  or  specific  social  handicaps  of  entrants  to  train- 
ing, or  students,  with  the  objective  of  changing  acceptance  patterns 
of  candidates  to  training  and,  both  efficiency  and  effectiveness  of 
the  training  administered. 

• Washouts  and  Dropouts.  The  innovation  may  aim  at  the  reclamation  of 
actual  and  potential  washouts  and  dropouts  from  training  programs. 

• Student  Evaluation.  Procedures  for  capturing,  organizing,  inter- 
preting, and  using  various  means  of  assessing  student  performance, 
during  and  at  the  end  of  training,  for  purposes  other  than  directly 
concerned  with  the  training  process  itself. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  DESCRIBE  EMPIRICAL  DATA  ON  BENEFITS  AND  SAVINGS 


DESCRIPTION 

The  innovation  may  have  been  subjected  to  pilot  testing  with  live  subjects 
under  experimental  or  demonstration  conditions.  Presumably,  the  study  has 
been  written  according  to  the  usual  requirements  for  replicability  of  what 
was  done.  All  reports  dealing  with  data  obtained  from  the  innovation  in  any 
kind  of  use  or  investigatory  tryout  should  be  Included  as  part  of  this  sub- 
task documentation. 

If  the  innovation  is  directed  principally  to  a cost  reduction  ip  a type  of 
instructional  vehicle  or  medium,  or  to  an  instructional  function,  cost  data 
based  on  eventual  procurement  costs,  plus  operation  and  maintenance  costs, 
should  be  obtained  as  part  of  the  present  documentation. 

PROCEDURE 

1.  Obtain  and  set  down  data  from  the  studies  that  show  quantitative  benefits 
in  the  learning  process,  or  products  of  learning  as  characterized  in 
Subtasks  01.02,  01.03,  and  01.04.  Note  any  qualifications  that  should  be 
applied  to  these  benefits  such  as  the  distributions  of  scores  around 
mean  results,  special  conditions  under  which  the  study  was  performed, 
size  and  nature  of  the  sample  of  subjects,  and  so  on.  These  benefits 
should  be  compared  with  respect  to  control  conditions  that  represent  a 
current  practice,  or  that  can  be  related  to  a current  training  practice. 
The  generalizations  made  In  the  conclusions  reported  about  the  study 
should  be  carefully  checked  against  the  actual  conditions  and  samplings 
In  the  study. 

• Criterion  Variables  for  Measuring  Training  Effectiveness.  If  the 
innovation  is  directed  primarily  to  a student  performance  benefit, 
the  experimental  conditions  may  enable  the  benefit  to  be  characterized 
by  one  or  more  of  the  following. 

a.  Parsimony  In  what  Is  learned,  with  respect  to  a given  level  and 
range  of  acceptable  task  performance,  or  its  equivalent  in  the 
expression  of  "knowledge."  This  is  the  minimum  range  and  level 
of  learning  that  enables  reliability  and  competence  in  all 
criteria  of  effective  job  task  performance.  Thus,  it  is  one 
measure  of  the  efficiency  of  training;  only  that  which  contributes 
to  a performance  criterion  is  learned. 
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b.  Validity  and  completeness  in  level  of  proficiency.  The  innovation 
may  be  directed  towards  increasing  the  job  task  relevance  of  what 
is  learned,  or  to  higher  levels  of  proficiency  than  is  practicable 
with  current  training  practices. 

c.  Completeness  in  range  of  what  is  learned.  The  innovation  may 
enable  kinds  of  skill  to  be  acquired  that  heretofore  were  not 
feasible  other  than  in  operational  situations;  or  the  innovation 
may  identify  skill  requirements  heretofore  neglected  in  training, 
but  which  contribute  to  overall  job  effectiveness. 

d.  Extensibility  of  skill  in  degree  or  in  kind.  If  the  Innovation 
enables  the  student  to  increase  his  skill  level  more  effectively 
by  on-the-job  experiences  as  a function  of  the  kind  of  training 
he  gets,  the  innovation  would  be  classified  under  this  rubric 
rather  than  b.  above.  The  innovation  may,  however,  demonstrably 
facilitate  the  acquisition  of  associated  skills  relative  to  the 
incumbent's  assigned  job,  or  related  jobs,  as  in  cross-training, 
or  for  a higher  level  job. 

e.  Work  motivation.  The  innovation  may  have  a primary  thrust  In  the 
increase  of  the  student's  motivation  to  do  the  work,  do  it  well, 
or  be  willing  to  perform  effectively  in  potentially  incapacitating 
stress  conditions.  This  motivation  should  be  differentiated  from 
willingness  to  accept  a given  billet  or  job  position. 

2.  Obtain  and  set  down  the  quantitative  costs  reported  in  the  studies  (or 
based  on  further  inquiry)  as  measured  by  the  relevant  input  conditions. 
Costs  may  be  expressed  in  comparative  practice  time  or  exposure  time,  or 
in  dollar  cost  per  unit  of  learning  per  student  if  the  innovation  is 
primarily  an  instructional  vehicle  function.  In  the  latter  case,  determine 
or  estimate  the  eventual  unit  procurement  cost  plus  estimated  costs  of 
operation  and  maintenance  (per  year)  of  the  device  or  medium.  Set  down 
any  factors  that  may  qualify  these  estimates. 

3.  Estimate  the  magnitudes  of  benefits  and  savings  under  actual  conditions 
in  the  Navy  context.  This  is  the  translation  of  the  findings  in  the 
study  to  values  in  benefits  and  savings,  or  both,  in  actual  training 
environments  and  performance  environments.  This  translation  is  a judg- 
mental process.  The  rationale  for  extrapolating  magnitudes  in  kind  and 
amount  from  the  study  to  real  life  should  be  made  explicit.  The  data 
derived  in  this  step  should  be  identified  and  differentiated  from  the 
data  reported  in  step  2. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 


SUBTASK:  ESTIMATE  THE  PRACTICAL  IMPORTANCE  OF  THE  TRAINING  PRODUCT 

RESULTING  FROM  THE  INNOVATION 


DESCRIPTION 

The  innovator  may  have  performed  or  referenced  inquiry  into  the  importance  of 
the  instructional  objectives  towards  which  the  innovation  is  directed.  Such 
information  can  assist  in  evaluating  the  potential  contribution  of  the 
innovation. 

PROCEDURE 

1.  Describe  or  estimate  the  practical  importance  of  the  training  product  which 
results  from  the  effects  of  the  innovation.  This  may  be  a gross  estimate 
of  the  importance  and  centrality  of  the  knowledge  or  skill  to  effective 
job  performance  as  a contribution  to  the  success  of  the  Navy  mission. 

The  training  outcome  to  which  the  innovation  is  relevant  may  be  critical, 
important,  potentially  useful,  or  irrelevant  to  mission  effectiveness  - 
that  is,  in  the  mainstream  of  job  actions  such  as  landing  an  aircraft,  or 
peripheral  to  the  mainstream  such  as  the  legibility  of  handwriting  In 
filing  a flight  report.  If  actual  on  the  job  data  are  available  (such  as 
accident  reports,  specific  kinds  of  failures  in  diagnostic  operations, 
procedural  inadequacies,  and  so  on)  that  can  focus  and  quantify  the  defi- 
ciency towards  which  the  innovation  is  aimed,  such  data  can  materially 
assist  the  evaluation. 

2.  The  empirical  benefits  plus  the  practical  importance  of  the  innovation  may 
be  sufficiently  clear-cut  to  justify  going  directly  to  Task  6 (Cost-Benefits 
Analysis),  and  an  acceptance  decision. 
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TASK:  FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 

SUBTASK:  ESTIMATE  RESOURCE  REQUIREMENTS  NEEDED  FOR  FURTHER  STUDY  AND 

IMPLEMENTATION  OF  THE  INNOVATION 


DESCRIPTION 

This  task  is  performed  on  the  assumption  that  further  research  and  development 
activity  may  be  required  to  bring  the  innovation  to  a state  of  implementation 
on  a practical  scale  and  in  real  training  situations.  It  may  also  be  assumed 
as  background  for  this  step  that  further  testing  of  the  effectiveness  of  the 
innovation,  such  as  in  experiments  in  various  settings  with  various  samples 
of  conditions  and  subjects,  may  be  necessary  or  at  least  desirable  for  making 
firm  commitment  about  the  benefits  and  range  of  benefits  to  be  derived  from 
the  innovation,  as  well  as  to  expose  its  liabilities. 

PROCEDURE 

1.  Provide  a breakdown  estimate  in  dollars,  time,  and  human  talent  of  the 
resources  expended  thus  far  in  developing  the  innovation  and  in  testing 
its  effectiveness. 

2.  With  step  1 as  a reference,  prepare  an  estimate  in  dollars,  time,  and 
human  talent  for: 

a.  Further  research  and  development  effort  required  to  bring  the  innova- 
tion to  level  of  development  where  it  can  be  introduced  and  used 
effectively  on  the  scale  and  in  the  contexts  of  practical  application. 

b.  Further  tryout  testing  and  empirical  data  collection  on  the  magni- 
tude and  range  of  effectiveness  of  the  innovation  that  can  assure 
better  than  a 90%  probability  that  the  projected  benefits  will  be 
achieved  and  the  projected  liabilities  kept  within  their  estimated 
bounds . 
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TASK:  DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION 

The  first  embodiment  of  an  innovation  may  incorporate  the  concept  but  lack 
implementation  know-how.  A few  hours  of  directed  effort  in  a selected  group 
of  experts  can  result  in  alternative  embodiments  that  may  sacrifice  little  of 
the  benefits  of  the  innovation  but  achieve  them  at  costs  greatly  below  those 
implied  by  the  original  embodiment.  It  may  also  turn  out  that  the  innovation 
is  not  an  innovation  in  fact  but  only  in  its  form  of  presentation,  and  that 
the  concept  has  been  previously  rejected  in  fact  and  in  principle,  or  that  it 
is  embodied  in  existing  practices  but  with  a different  name  and  appearance. 

The  search  for  alternative  embodiments  does  imply  that  the  functional  principle 
and  objectives  contained  in  the  innovative  concept  are  understood  and  transmitted. 

Alternatives  may  also  be  found  by  searching  the  literature  or  by  redirecting 
the  expertise  of  the  innovator. 

General  knowledge  in  the  subject  field,  plus  common  sense,  may  direct  efforts 
at  gross  cost  reduction  on  the  more  cost-sensitive  variables.  A cost-sensitive 
variable  is  one  that  has  a large  multiplier  factor;  it  can  also  be  a cost 
factor  with  many  cost  dependencies. 

The  level  of  human  expertise  required  to  effectively  implement  the  innovation 
in  training  practices  can  be  a crucial  cost  factor,  not  readily  translated  into 
dollars  but  central  to  the  practicality  of  the  innovation.  This  is  especially 
true  if  any  exercise  of  the  Innovation  in  the  real  world  depends  on  a high  level 
of  creative  or  logical  talent. 

On  the  other  hand,  an  innovation  that  may  be  impractical  at  one  time  or  in  one 
context  may  be  highly  serviceable  in  other  contexts.  Failure  history  should, 
therefore,  be  examined  for  its  relevance  to  current  or  projected  needs  and 
opportunities.  For  example,  a simulation  function  that  was  impractical  for 
analog  computers  may  be  trivial  in  cost  on  a contemporary  digital  computer. 

A commonsense  search  for  functional  alternatives  and  their  evaluation  may  save 
extensive  time  and  effort  in  putting  the  original  statement  of  the  innovation 
through  the  assessment  mill.  A potential  source  of  alternatives  can  be  the 
archives  of  innovations  that  were  proposed  in  the  past  but  rejected  for  various 
reasons  applicable  at  the  time  but  no  longer  applicable. 

Any  alternative  to  the  innovation  as  it  was  originally  proposed  should  be 
described  in  the  same  format  that  was  used  to  describe  the  original  proposal 
(Task  1). 
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1.  Consult  sources  for  developing  alternatives  to  the 
innovation. 

2.  Develop  and  describe  alternatives  to  the  innovation. 

3.  Make  an  intuitive  assessment  of  the  alternative. 
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TASK:  DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION 


SUBTASK:  CONSULT  SOURCES  FOR  DEVELOPING  ALTERNATIVES  TO  THE  INNOVATION 


PROCEDURE 

1.  Consult  available  sources  of  information  and  ideas  leading  to  the  develop- 
ment of  alternatives  to  the  innovation  that  may  be  potentially  more  feasible, 
practical,  or  otherwise  have  more  benefits  and  fewer  liabilities  than  the 
innovation  as  proposed. 

Potential  sources  for  alternatives  may  be  found  in  the  following  ways: 

• Understanding  the  objectives  of  the  innovation  so  clearly  that  some 
aspects  of  the  innovator's  implementation  become  recognized  as  non- 
essential  or  irrelevant  to  the  central  objective  of  the  innovation. 
Stripping  off  the  appurtenances  from  the  innovation  as  originally 
proposed  may  make  it  more  practical  and  feasible  to  accomplish. 

• Reviewing  the  literature  on  the  topic  with  which  the  innovation  deals. 
Alternatives  may  already  exist,  and  data  may  be  available  about  them. 

• Finding  experts  on  the  topic  and  consulting  with  them,  getting  their 
suggestions  about  alternatives  to  the  innovation  as  proposed.  Experts 
may  also  be  used  to  make  preliminary  reviews  and  technical  assessments 
of  alternatives  to  the  innovation  when  these  alternatives  are  developed. 
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TASK:  DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION 


SUBTASK:  DEVELOP  AND  DESCRIBE  ALTERNATIVES  TO  THE  INNOVATION 


DESCRIPTION 

This  process  should  be  no  more  formal  or  time-consuming  than  is  dictated  by 
common  sense  applied  to  the  alternatives.  Some  ideas  as  alternatives  can  be 
evaluated  and  dismissed  in  the  course  of  casual  conversation.  Others  may 
justify  no  more  than  a few  notes  of  description  that  may  be  sufficient  to 
find  a key  weakness  that  eliminates  the  alternative  from  further  examination. 

An  alternative  that  survives  the  Incubation  and  early  critical  processes, 
should  be  set  down  in  sufficient  detail  so  that  if  It  fails  in  later  stages 
of  even  informal  assessment,  it  can  be  put  into  the  data  base  of  "Innovations." 

PROCEDURE 

1.  Prepare  a description  of  the  alternative  to  the  innovation  as  proposed 
by  the  Innovator.  The  description  should  be  formatted  according  to  the 
outline  in  Task  1.  Go  no  further  in  detail  than  seems  justified  by  the 
apparent  quality  of  the  alternative. 

2.  If,  as  a consequence  of  interplay  between  the  inventive,  descriptive,  and 
critical  processes  in  the  development  of  the  alternative,  it  becomes  a 
promising  candidate  for  competing  with  the  Innovation  as  proposed,  a formal 
description  should  be  made  of  it.  Follow  the  format  described  in  Task  1. 
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TASK:  DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION 


SUBTASK:  MAKE  AN  INTUITIVE  ASSESSMENT  OF  THE  ALTERNATIVE 


DESCRIPTION 

The  formal  description  of  the  alternative  may  make  apparent  some  difficulties 
that  were  obscure  before  that  description.  The  formal  description  also 
enables  the  assessor  and  his  associates  to  back  off  from  the  statement  of  the 
alternative  as  they  did  from  the  original  description  of  the  innovation  by  the 
innovator.  The  question  can  be  asked,  globally  and  analytically;  "Does  this 
thing  make  sense?  Where's  the  fundamental  flaw  in  assumption  or  procedure? 

Does  the  proposed  implementation  really  lead  to  the  objective  stated?  Is  the 
objective  itself  stated  operationally  enough  that  one  could  tell  whether  it 
was  met  or  not  met?"  and  so  forth. 

PROCEDURE 

1.  Make  an  intuitive  assessment  of  the  description  of  the  alternative. 

2.  If  the  result  of  the  intuitive  assessment  is  favorable,  complete  the  formal 
description  of  the  alternative.  It  should  then  be  put  through  the  ETAM 
procedural  model;  Tasks  3 through  8.  If  the  result  is  unfavorable,  put 
the  description  and  the  assessment  into  the  archives. 

3.  The  alternative  to  the  original  innovation  may  not  have  equivalent  objectives 
and  functional  capabilities.  If  so,  the  alternative  must  have  its  own 
analysis  made,  not  only  through  Tasks  3 and  4,  but  also  through  the  range- 
of-effect  and  benefits/liabilities  assessments  in  Task  5.  It  is  unnecessary 
for  the  alternative  to  have  objectives  equivalent  to  the  innovation  in  order 
for  it  to  be  subject  to  comparative  analysis. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


The  assessment  and  decision  process  relative  to  the  educational  innovation 
and  identified  alternatives  began  in  Tasks  1 and  2 with  rather  macro-type 
judgments.  Task  3 begins  the  more  formal  aspect  of  the  assessment  and  deci- 
sion process  which  is  further  refined  in  Task  4 (Perform  Analytic  Feasibility 
Assessment).  The  outputs  of  these  tasks  become  the  basis  for  making  the  in- 
depth  assessment  outlined  in  Task  6 (Perform  Cost-Benefits  Analysis)  and  in 
Task  7 (Perform  Financial  Analysis). 

The  assessment  process  must  consider  the  proposed  innovation  and  alternatives 
in  two  contexts.  The  first  is  in  terms  of  its  intrinsic  benefits.  Improved 
training  performance,  decreased  course  length,  reduced  costs,  etc.,  are  typical 
examples  of  intrinsic  benefits  resulting  from  application  of  the  innovation. 
Quantification  and  aggregation  of  these  benefits  will  be  a major  part  of  the 
total  assessment  procedure.  The  second  context,  however,  relates  to  more 
global  concerns  that  may  impact  the  development,  implementation,  or  ultimate 
acceptance  by  the  using  system.  Organizational  compatibility,  funding  avail- 
ability, attitudinal  acceptance,  etc.,  are  typical  factors  with  which  the 
assessor  is  concerned  at  this  point  in  the  assessment  process.  A framework 
for  viewing  these  two  assessment  contexts  can  be  seen  by  referring  to  the  decisi 
tree  in  Figure  3.0.1.  The  intrinsic  benefits  are  aggregated  for  each  of  the 
important  outcomes,  A through  D.  The  second  group  of  factors,  and  the  ones 
to  be  considered  here  and  in  Task  4,  will  be  used  to  establish  the  proba- 
bilities at  points  U through  Z.  The  overall  procedure  will  be  detailed  in 
Task  4;  however,  at  this  point  the  intent  is  to  check  the  innovation  for 
reasonability  in  the  context  of  the  factors  which  may  affect  the  probability 
estimates  of  implementation  success  and  using  system's  acceptance. 


SUBTASK 


REFERENCE 


1.  Importance  of  the  Innovation  to  the  Projected  Navy  03.01.01 

Mission.  Make  a gross  estimate  of  the  potential  contri- 
bution of  the  innovation  (if  successfully  implemented) 

to  the  support  of  specific,  identified  elements  in  the 
Navy  mission.  These  elements  may  lie  within  the  training 
mission,  in  the  operational  mission,  or  in  a relationship 
between  the  training  and  operational  missions. 

2.  Organizational  Compatability.  Estimate  the  degree  to  03.02.01 

which  the  effects  of  the  innovation  fit  in  with  command 

and  career  organizational  structures  of  jobs  and  duties 
and  policies  for  the  use  of  personnel. 

3.  Goals/Policy  Compatability.  Estimate  the  degree  to  which  03.03.01 
the  proposed  innovation  fits  with  existing  goals  and 

policies  within  the  training  and  other  personnel  organiza- 
tions where  implementation  is  expected  to  take  place. 
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A SAVINCS/BCNEFlTS 
ACCRUE 

B COSTS -POSSIBLE 
LIABILITIES  RESULT 

C COSTS  ACCRUE 

O NEUTRAL  OUTCOME 

A SAVINGS/BENEFITS 
ACCRUE 

B COSTS  POSSIBLE 
LIABILITIES  RESULT 

C COSTS  ACCRUE 

O NEUTRAL  OUTCOME 

A SAVINGS/BENEFITS 
ACCRUE 

B COSTS  POSSIBLE 
LIABILITIES  RESULT 

C COSTS  ACCRUE 

0 NEUTRAL  OUTCOME 


FIGURE  3.0.1.  DECISION  TREE  - A FRAMEWORK  FOR  ASSESSING 
TRAINING  INNOVATIONS 
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4.  State-of-the-Art.  Make  an  estimate  of  the  technical 
feasibility  of  implementing  the  proposed  innovation  on 
the  scale  necessary  for  practical  application.  Consider 
any  unique  technical  requirements  that  need  extensive 
R&D  effort  to  accomplish. 

5.  Funding.  Make  an  estimate  of  the  magnitude  of  dollar 
support  fcr  further  R&D  effort  in  bringing  the  proposed 
innovation  to  the  point  of  actual  operational  introduction 
and  application.  These  costs  should  include  those  to  be 
spent  in  modifying  existing  facilities  and  practices  in 
order  to  accept  the  innovation,  and  costs  from  retraining 
persons  required  to  implement  the  innovation. 

6.  Technical  Support.  Estimate  the  extent  to  which  qualified 
personnel  of  whatever  kinds  are  needed  can  be  made  avail- 
able to  support  further  R&D  or  implementation  of  the  innova- 
tion or  maintenance  of  the  innovation  in  practice.  High 
levels  of  specialized,  generalized,  or  creative  talent 
(especially  the  latter)  are  hard  to  find,  and  many  demands 
compete  for  their  services. 

7.  Attitudinal  Acceptance.  Estimate  the  degree  to  which  the 
instructional  community  and/or  the  students  will  accept 
the  device  or  function  as  desirable,  meaningful,  and 
appropriate  to  their  sense  of  role  and  self-image. 

8.  Summarize  the  Preliminary  Feasibility  Profile  Data. 

Identify  risk  by  category,  numbers  of  risk  reduction 
projects,  and  estimated  costs  for  these  projects. 

RATIONALE 

The  innovation  may,  at  this  stage,  show  up  so  many  major  impediments  that  the 
assessor  feels  justified  in  recoronending  a rejection  of  the  proposal  to  the 
decision  maker  without  further  analysis.  If  this  is  not  the  case,  the  data 
developed  here  will  feed  into  Task  4 (Perform  Analytic  Feasibility  Assessment) 
and  .later  into  Task  6 (Perform  Cost-Benefits  Analysis)  and  Task  8 (Make  Accept/ 
Reject/Study  Decision). 

COMMENT 

The  assessor  is  cautioned  here  on  a decision  to  reject  without  an  explicit  or 
intuitive  understanding  of  the  range  of  applications  for  the  innovation.  Be- 
cause virtually  every  major  innovation  at  its  inception  has  presented  almost 
insuperable  obstacles,  shortsightedness  or  conservatism  at  this  point  may  cause 
premature  rejection  of  a potentially  valuable  enhancement  to  training. 


03.07.01 


03.08.01 


00  03 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 

SUBTASK:  ESTIMATE  THE  IMPORTANCE  OF  THE  INNOVATION  TO  THE  PROJECTED  NAVY 

MISSION 


PROCEDURE 

1.  Estimate  the  overall  significance  of  the  innovation  and  its  implications 
for  the  goals  and  mission  of  the  Navy.  These  may  consist  of  the  mission 
of  training,  operations,  or  of  the  relationship  between  the  two.  The 
estimate  should  take  into  account  future  trends  and  demands  and  restric- 
tions that  may  be  placed  on  Navy  institutions  in  terms  of  student  and 
instructor  input,  turnaround  time  in  preparing  and  administering  training, 
flexibility,  and  similar  issues.  The  potential  cost  advantages  of  the 
innovation  should  be  treated  as  secondary  in  arriving  at  a judgment  of  the 
functional  importance  of  the  innovation. 

2.  Select  the  statement  below  which  best  describes  the  estimated  importance 
of  the  innovation  if  it  were  implemented  and  widely  applied  to  its 
potential  range  of  usefulness. 

( ) The  innovation  would  have  an  effect  of  little  or  trivial  impor- 

tance to  the  overall  training  and  operational  missions  of  the 
Navy. 

( ) It  would  have  an  effect  of  moderate  importance. 

( ) Its  effect  would  be  substantial  and  important. 

( ) Its  effect  would  be  highly  important  and  possibly  crucial  to 

training  and/or  operational  missions  of  the  Navy. 

3.  Decide  if  the  innovation  has  important  enough  application  within  the  Navy 
to  justify  pursuing  the  analysis.  This  consideration  should  be  extended 
into  future  years  where  the  possibility  of  changing  missions  may  change 
the  importance  estimate. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  OF  ORGANIZATIONAL  COMPATABILITY 


PROCEDURE 

1.  Consider  the  degree  to  which  the  effects  that  may  be  produced  by  the 
innovation,  or  that  may  be  involved  in  its  use,  conflict  with  or  fit 
into  comnand  and  career  organizational  structures  of  jobs  and  duties 
and  policies  for  the  use  of  personnel.  This  judgment  should  not  be 
affected  by  whether  the  assessor  believes  the  outcome  from  the  innova- 
tion would  be  desirable  or  undesirable. 

2.  Select  the  statement  below  which  best  fits.  The  innovation, would  lead 
to  consequences  that  would  be: 

( ) Incompatible  and  inconsistent  with  Navy  conmand  and  career 

structures  as  they  exist  or  are  planned. 

( ) Somewhat  inconsistent  with  Navy  conmand  and  career  structures. 

( ) Irrelevant  to  Navy  conmand  and  career  structures. 

( ) Supportive  of  Navy  conmand  and  career  structures. 

3.  Decide  whether  the  risks  are  so  high  that  no  viable  set  of  implementation 

projects  could  be  described  to  reduce  them  to  an  acceptable  level.  If 
this  is  the  case,  then  rejection  should  be  considered  at  this  point.  If 

potentially  high  risks  exist,  then  thought  should  be  given  at  this  time  to 

risk  reduction  projects  to  be  detailed  in  Task  4.  If  the  projected  project 
cost  seems  unreasonably  high  in  relation  to  the  benefits  associated  with 
the  reduction  of  risk,  then  this  is  perhaps  another  reason  for  rejection 

at  this  time. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  OF  GOALS/POLICY  COMPATIBILITY 


PROCEDURE 

1.  Consider  the  degree  to  which  the  proposed  innovation  fits  with  existing 

goals  and  policies  within  the  training  and  other  personnel  organizations, 
where  Implementation  Is  expected  to  take  place.  Thus,  for  example,  the 
goal  of  the  training  establishment  could  be  the  training  of  the  person 
not  only  for  a specific  military  job  or  position,  but  for  any  of  a 
family  of  jobs,  or  for  a counterpart  in  the  civilian  specialty  as  well 

as  the  military  job.  It  is  assumed  that  the  assessor  is  familiar  with 

those  goals  and  policies  that  could  be  applicable  to  the  innovation. 

2.  Select  the  statement  below  which  best  fits  the  case.  The  innovation 
would  lead  to: 

( ) Substantial  conflicts  with  existing  goals  and  policies  of  one 

or  more  training  and  other  personnel  organizations. 

( ) Minor  conflicts  with  those  policies. 

( ) No  conflicts. 

( ) Support,  in  general,  of  existing  goals  and  policies. 

3.  As  in  the  preceding  procedure,  an  assessment  of  risks,  risk  reduction 
projects,  and  the  apparent  cost-benefits  of  these  projects  should  be 
made  to  determine  if  the  innovation  should  be  rejected  at  this  time. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  OF  THE  STATE-OF-THE-ART  APPLICABLE  TO  THE  INNOVATION 


PROCEDURE 

1.  Consider  the  technical  feasibility  of  the  proposed  Innovation  In  terms 
of  the  scope  and  scale  that  would  make  Its  Implementation  practical. 

Take  into  account  unique  technical  requirements  that  might  need  exten- 
sive R&D  effort  to  accomplish,  In  order  to  develop  and  Integrate  the 
Innovation  into  an  Instructional  or  operational  system. 

2.  Select  the  statement  below  which  best  fits  the  case.  The  Innovation  has: 

0 

( ) Low  probability  of  success  In  overcoming  technical  difficulties, 

even  with  substantial  continued  R&D  effort. 

( ) Moderate  probability  of  technical  success  with  some  continued 

R&D  effort. 

( ) High  probability  of  success  with  at  least  moderate  R&D  support. 

( ) Amply  demonstrated  technical  feasibility  at  this  time. 

3.  Decide  whether  the  risks  are  so  high  that  no  viable  set  of  Implementation 
projects  could  be  described  to  reduce  them  to  an  acceptable  level.  If 
this  is  the  case,  then  rejection  should  be  considered  at  this  point.  If 
potentially  high  risks  exist,  then  thought  should  be  given  at  this  time  to 
risk  reduction  projects  to  be  detailed  In  Task  4.  If  the  projected  project 
cost  seems  unreasonably  high  In  relation  to  the  benefits  associated  with 
the  reduction  of  risk,  then  this  Is  perhaps  another  reason  for  rejection 

at  this  time. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  FURTHER  R&D  FUNDING 


PROCEDURE 

1.  Examine  the  number  and  kinds  of  R&D  effort  still  required  to  be  completed 
in  order  to  bring  the  innovation  to  completion  — that  is,  to  actual  opera- 
tional introduction  and  application.  The  R&D  may  be  in  terms  of  systems 
design,  hardware,  software,  communications,  and  other  factors.  (See  Task  5, 
Determine  Range-of-Effect,  Subtask  2,  Develop  Background  for  Instructional 
Vehicle  Analysis,  for  a list  of  potentially  relevant  usage  properties.) 

The  costs  of  technical  and  operational  debugging  of  new  technologies  should 
not  be  underestimated.  Costs  required  to  modify  existing  facilities  and 
practices  in  order  to  accept  the  innovation  should  be  included,  as  well  as 
costs  from  retraining  persons  required  to  implement  the  innovation.  It  may 
be  sufficient  to  make  gross  estimates  of  potential  costs  for  various 
elements  of  the  expected  R&D  effort. 

2.  Estimate  the  relative  magnitude  of  the  expected  R&D  costs.  This  estimate 
should  reference  the  available  or  expected  overall  R&D  budget  of  the  Navy 
for  training.  Select  the  statement  which  best  fits  the  case. 

If  a comparison  is  made  with  the  typical  size  of  funding  for  an  R&D 
objective  in  Navy  education  and  training,  the  R&D  expenditure 
realistically  to  be  expected  in  order  to  bring  the  proposed  innova- 
tion into  operational  implementation  will  be: 

( ) Large  to  very  large  to  bring  the  Innovation  to  an  operational 

level  of  implementation. 

( ) Moderate. 

( ) Readily  integrated  into  existing  planned  programs  of  expenditures 

for  training  R&D. 

( ) An  inmediate  and  relatively  direct  savings  to  the  R&D  budget  for 

training. 

3.  Decide  whether  the  costs  are  so  high  to  preclude  obtaining  adequate  funding. 
Alternative  approaches  at  a lesser  level  of  funding  should  be  considered  at 
this  point.  If  potentially  high  risks  exist,  then  thought  should  be  given 
at  this  time  to  possible  rejection.  If  the  projected  R&D  cost  seems  un- 
reasonably high  in  relation  to  the  benefits  associated  with  the  reduction 
of  risk,  then  this  is  perhaps  another  reason  for  rejection  at  this  time. 


Ill -43 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
03  05  02 


4.  Retain  any  estimates  of  actual  dollar  costs  that  were  developed  in  this 
subtask.  Document  these  data  in  the  project  file. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  REQUIRED  TECHNICAL  SUPPORT 


PROCEDURE 

1.  Examine  the  various  kinds  of  skills  and  abilities  that  will  be  needed  in 
order  to  bring  the  proposed  innovation  into  service.  These  may  require 
those  skills  needed  for  the  R&D  work  and  also  those  skills  that  will  be 
necessary  to  maintain  and  apply  the  innovation,  assuming  it  has  been 
implemented.  Then,  assess  the  ease  or  difficulty  in  finding  people  with 
these  skills  and  skill  levels  in  the  quantity  that  may  be  needed.  It 
should  be  recognized  that  novel  creative  skills  are  always  in  short 
supply  and  heavy  demand,  especially  when  various  technical  constraints 
may  be  placed  upon  their  talent. 

2.  Select  the  statement  below  which  best  fits  the  estimate  of  level  of 
technical  support  that  will  be  required  to  develop  and  maintain,  in 
practice,  the  innovation: 

( ) Extremely  rare  kinds  of  talent  will  be  required  to  (1)  continue 

the  R&D  or;  (2)  to  implement  the  innovation  in  practice  or; 
both  (1)  and  (2)  apply. 

( ) Some  difficulty  should  be  expected  in  finding  the  kind  of 

talent  needed  to  complete  the  R&D  work  and  to  implement 
the  innovation  in  practice. 

( ) Adequate  talent  is  available  or  can  be  made  available  for 

bringing  the  innovation  to  practical  and  realistic  imple- 
mentation in  training. 

( ) Adopting  the  innovation  will  actually  reduce  the  level  of 

talent  now  required  for  performing  the  function  intended 
for  the  innovation  in  the  training  subsystem. 

3.  Decide  whether  the  risks  are  so  high  that  no  viable  set  of  implementation 
projects  could  be  described  to  reduce  them  to  an  acceptable  level.  If 
this  is  the  case,  then  rejection  should  be  considered  at  this  point.  If 
potentially  high  risks  exist,  then  thought  should  be  given  at  this  time  to 
risk  reduction  projects  to  be  detailed  in  Task  4.  If  the  projected  project 
cost  seems  unreasonably  high  in  relation  to  the  benefits  associated  with 
the  reduction  of  risk,  then  this  is  perhaps  another  reason  for  rejection 

at  this  time. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  ESTIMATE  ATTITUDINAL  ACCEPTANCE 


PROCEDURE 

1.  On  the  basis  of  historical  precedents  or  current  expressions  of  attitude 
about  issues  represented  by  the  innovation,  determine  the  degree  to  which 
users  of  the  innovation  are  likely  to  accept  the  innovation  as  desirable, 
meaningful,  and  appropriate  to  their  sense  of  role  and  of  self-image. 
General  experience  has  shown  that  if  the  instructor  rejects  the  instruc- 
tional device,  function,  or  role,  he  will  transmit  this  attitude  to  most 
of  his  students.  This  does  not  mean  that  the  converse  occurs,  namely 
that  instructor  acceptance  is  tantamount  to  student  acceptance.  The 
assessor  should  recognize  that  some  resistance  may  be  either  transient 

or  a deep  and  long-lasting  resentment. 

2.  Select  the  statement  below  which  best  describes  the  kind  of  attitudinal 
acceptance  to  be  expected  if  the  innovation  is  introduced: 

( ) The  instructional  community  will  almost  certainly  respond 

to  the  innovation  with  sustained  hostility. 

( ) The  instructional  community  will  probably  respond  with  some 

moderate  but  temporary  resistance  to  the  innovation. 

( ) The  innovation  can  be  introduced  to  the  instructional 

community  with  little  or  no  feeling  about  it,  one  way  or 
the  other. 

( ) On  the  main,  the  instructional  community  will  welcome  the 

innovation,  especially  after  its  realistic  values  have  been 
experienced. 

3.  Decide  whether  the  risks  are  so  high  that  no  viable  set  of  implementation 
projects  could  be  described  to  reduce  them  to  an  acceptable  level.  If 
this  is  the  case,  then  rejection  should  be  considered  at  this  point.  If 
potentially  high  risks  exist,  then  thought  should  be  given  at  this  time  to 
risk  reduction  projects  to  be  detailed  in  Task  4.  If  the  projected  project 
cost  seems  unreasonably  high  in  relation  to  the  benefits  associated  with 
the  reduction  of  risk,  then  this  is  perhaps  another  reason  for  rejection 

at  this  time. 
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TASK:  MAKE  PRELIMINARY  FEASIBILITY  PROFILE 


SUBTASK:  SUMMARIZE  PRELIMINARY  FEASIBILITY  PROFILE  DATA 


PROCEDURE 

1.  Assemble  the  data  from  Task  3,  Subtasks  1 through  7 into  the  format 
suggested  in  Figure  3.8.1.  Check  the  level  of  risk  and  importance  without 
considering  additional  risk  reduction  projects. 

2.  Indicate  the  number  of  risk  reduction  projects  required  and  identified 
for  each  risk  category. 

3.  Estimate  the  approximate  total  cost  for  risk  reduction  projects  in  each 
category. 

4.  Sunmarize  the  risk  data  with  an  overall  risk  assessment,  total  for  risk 
reduction  projects,  and  estimated  total  cost  for  the  projects. 

5.  An  extremely  useful  exercise  at  this  point  would  be  to  reevaluate  the 
individual  risk  levels  considering  successful  completion  of  all  the 
identified  risk  reduction  projects. 

COMMENT 

The  data  sunmarized  here  may  prompt  a rejection  decision  at  this  point,  how- 
ever, whether  a rejection  occurs  or  analysis  continues,  the  data  will  be  a 

valuable  input  to  the  management  decision  process. 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

Having  decided  as  the  result  of  Task  3 that  there  are  no  insurmountable 
obstacles  in  achieving  the  benefits  of  the  proposed  innovation,  more  rigorous 
application  of  the  factors  outlined  in  Task  3 will  be  made  at  this  stage  of 
the  assessment  process.  This  task  has  a two-fold  purpose.  First  is  to  refine 
the  analysis  of  Task  3 in  developing  success  probabilities  for  inclusion  in 
the  decision  tree.  Second  is  to  further  define  implementation  projects 
targeted  at  reducing  the  risk  from  any  of  the  factors  considered  in  Task  3. 
Once  risk  reduction  programs  have  been  defined,  the  success  probabilities  are 
reestimated  and  used  along  with  the  original  probability  estimates  to  complete 
the  decision  tree.  If  no  major  problem  areas  were  identified,  then  Task  4 
would  be  bypassed;  although  the  factors  considered  in  the  analysis  of  Task  3 
should  be  reassessed  after  Task  5 (Determine  Range-of-Effect)  has  been 
completed,  since  a broadened  application  of  the  innovation  may  expose  new 
problems. 

SUBTASK:  REFERENCE 

1.  Identify  the  factors  from  Task  3 that  represent  potentially  04.01.01 
high  risks  in  the  implementation  and  use  of  the  proposed 
innovation.  (This  and  the  following  subtasks  are  also 
applied  to  each  alternative  resulting  from  the  analysis  in 
Task  2.) 

2.  Evaluate  the  probabilities  that  the  innovation  can  be 
successfully  implemented  and  used  considering  the  identi- 
fied risks. 

3.  Define  projects  and  associated  costs  intended  to  reduce 
risks  to  a reasonable  level. 

4.  Reevaluate  the  probabilities  of  successful  implementation 
and  use  considering  achievement  of  project  results. 

5.  Make  preliminary  estimate  of  potential  scope  of  applica- 
tion ( ran ge-of-ef feet)  required  for  project  payoff. 

6.  Create  tickler  file  entry  identifying  criteria  for 
reassessment  of  innovation  if,  at  this  point,  the  pro- 
posal does  not  appear  feasible. 


04.02.01 

04.03.01 
04.04.01 

04.05.01 

04.06.01 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

SUBTASK:  IDENTIFY  POTENTIALLY  HIGH  RISK  AREAS 


PROCEDURE 

I'.  Refer  to  the  output  of  Task  3 which  identifies  risks  which  may  impact  the 
projected  implementation  or  use  of  the  proposed  innovation  in  the  follow- 
ing topical  categories. 

• Importance  of  innovation  to  projected  Navy  mission 
t Organizational  compatibility 

• Goals/policy  compatibility 

• State-of-art  (Technological  feasibility) 

• Funding  constraints 

• Technical  support  capability 

• Attitudinal  acceptance. 

2.  Develop  further  detail  for  each  risk  statement  which  describes  the  specific 
obstacle  to  be  overcome  (the  potential  problem  envisioned  in  implementing 
and/or  using  the  innovation).  The  detail  should  suggest  R&D  studies  or 
projects  which  can  be  undertaken  to  reduce  the  overall  risk  in  each  major 
category.  More  than  one  approach  to  reducing  the  risks  may  be  identified 
and  each  reasonable  alternative  should  be  described  and  carried  through  the 
subsequent  analysis. 

3.  Document  results  of  analysis  and  incorporate  into  project  file. 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

SUBTASK:  EVALUATE  SUCCESS  PROBABILITIES 


DESCRIPTION 

Evaluation  of  success  probability  is  performed  with  respect  to  implementation 
of  the  innovation  as  well  as  in  relation  to  its  acceptance  by  the  using  system. 
Some  of  the  risk  categories  from  the  preceding  subtask  will  affect  implementa- 
tion; e.g.,  state-of-art,  others  will  affect  innovation  use;  e.g.,  funding 
constraints.  In  this  procedure  the  risk  items  are  separated  so  that  an  inde- 
pendent evaluation  can  be  made  for  each  category.  The  process  outlined  in  this 
procedure  may  require  several  iterations  using  judgmental  inputs  from  a number 
of  experts  in  the  specific  problem  areas.  Also,  since  the  impacts  may  not  be 
Independent;  e.g.,  the  resolution  of  an  organizational  incompatibility  may 
change  the  probability  for  a technical  support  type  problem;  the  interrela- 
tionships and  conditional  probabilities  may  have  to  be  worked  out  using  extended 
decision  tree  diagrams  or  some  form  of  cross-impact  tables*.  The  basic  decision 
tree  format  discussed  in  Task  3 and  shown  in  Figure  4.2.1  will  be  used  as  a start- 
ing point. 

PROCEDURE 

% 

1.  Classify  the  risk  statements  from  04.01  into: 

a.  Those  that  will  create  problems  in  the  implementation  of  the  innovation. 

b.  Those  that  will  interfere  with  the  successful  use  of  the  innovation 
once  it  has  been  implemented. 

2.  Given  no  specific  project  action  is  undertaken  to  resolve  any  of  the  prob- 
lem areas  (other  than  a continuing  level  of  awareness  that  the  problem 
exists),  estimate  the  probability  that: 

a.  Successful  implementation  (includes  any  development  and  installation 
activities)  can  be  achieved  (U  in  Figure  4.2.1). 

. b.  The  user  will  accept  the  application  of  the  innovation  as  planned 

(V  in  Figure  4.2.1).  User  in  this  context  refers  to  the  using  system; 
i.e.,  organization,  function,  environment,  etc. 

COMMENT:  Composite  probabilities  where  the  problem  areas  are  not  independent 

may  require  more  complex  analysis  than  is  suggested  here. 


♦Paper  on  DELPHI  Techniques  and  Cross  Impact  Analysis  - Author  unknown. 
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3.  For  each  of  the  above  two  estimates,  provide  an  interval  estimate  where 
the  one  end  of  the  range  is  the  probability  of  success  under  the  most 
favorable  conditions  and  the  other  end  of  the  range  is  the  probability  of 
success  under  the  most  adverse  conditions. 

4.  Document  results  of  these  estimates  for  Incorporation  into  the  project 
file. 

RATIONALE 

The  probability  estimates  and  confidence  intervals  developed  in  this  subtask 
will  be  part  of  the  data  used  to  determine  the  relative  value  of  the  decision 
to  either: 

a.  Reject  the  proposed  innovation  outright. 

b.  Accept  it  without  additional  study. 

c.  Accept  It  and  conduct  continued  studies  considering  their  associated 
costs. 

The  confidence  intervals  will  be  used  in  conjunction  with  sensitivity  analyses 
to  determine  if  additional  information  should  be  acquired  in  order  to  Improve 
the  confidence  in  the  estimate. 
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FIGURE  4.2.1.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
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TASK:  * PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 
SUBTASK:  DEFINE  IMPLEMENTATION  PROJECTS  AND  COSTS 


DESCRIPTION 

The  assessor  must  decide  whether  the  level  of  risk  identified  is  potentially 
too  high  to  be  acceptable.  This  general  reference  resulted  in  the  previously 
identified  risk  statements.  The  task,  at  this  point,  is  to  define  specific 
projects  targeted  at  overcoming  some  of  the  problems  which  create  the  poten- 
tially high  level  of  risk,  and  to  cost  out  these  projects  for  inclusion  in 
the  decision  tree  outcomes  resulting  from  the  decision  to  carry  out  the  pro- 
jects. The  preliminary  cost  estimates  performed  in  Task  3 will  be  refined 
at  this  point. 

PROCEDURE 

1.  Prepare  a project  description  for  each  R&D  effort,  study,  experiment,  etc., 
which  should  be  considered  in  reducing  a specific  problem  or  group  of 
problems  to  a reasonable  level  of  risk. 

2.  Identify  resource  requirements  and  approximate  time  schedule  for  accom- 
plishing each  project. 

3.  Prepare  a detailed  cost  analysis  of  each  project.  A sample  form  for 
documenting  cost  data  is  shown  in  Figure  4.3.1. 

4.  Group  projects  into  one  or  more  packages,  each  of  which  is  intended  to 
reduce  the  overall  risk  to  a reasonable  level.  Where  alternative  approaches 
to  reducing  risk  have  been  defined,  groupings  of  projects  can  be  structured 
to  include  the  various  alternatives.  Different  cost/benefit  patterns  will 
undoubtedly  result  from  the  alternative  risk  reduction  packages  and  the 
effect  upon  the  decision  variable  will  be  determined  when  the  decision  tree 
is  folded  back  as  outlined  in  Subtask  04.05. 

5.  Document  results  of  project  definition  and  costing  for  incorporation  into 
the  project  file. 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

SUBTASK:  REEVALUATE  SUCCESS  PROBABILITIES 


DESCRIPTION 

This  involves  procedures  similar  to  those  outlined  in  Subtask  04.02. 

PROCEDURE 

1.  Given  that  the  specified  risk  reduction  projects  are  undertaken  for  each 
grouping  or  package  of  risk  reduction  projects,  reestimate  the  probability 
that  the  projects  will  succeed.  The  values  resulting  from  the  estimates 
will  be  assigned  to  the  "projects  succeed"  leg  on  the  decision  tree  (I  for 
project  package  1 in  Figure  4.2.1).  The  value  for  "projects  fail"  (1-1) 
is  also  entered  on  the  decision  tree.  If  more  than  one  project  package 
has  been  defined,  similar  entries  will  be  made  for  each  additional  package. 

2.  Given  that  the  projects  succeed,  for  each  project  package  estimate  the 
probability  that: 

a.  Successful  implementation  can  be  achieved  (W  in  Figure  4.2.1). 

b.  The  user  will  accept  the  application  of  the  innovation  as  planned 
(X  in  Figure  4.2.1). 

3.  Similarly,  given  that  the  projects  fail,  for  each  project  package  estimate 
the  probability  that: 

a.  Successful  implementation  can  be  achieved  (Y  in  Figure  4.2.1). 

b.  The  user  will  accept  the  application  of  the  innovation  as  planned 
(Z  in  Figure  4.2.1). 

C0WENT:  In  this  reassessment  of  risk,  consideration  shall  be  given  to  the 

time  required  to  complete  identified  projects  from  the  standpoint  that  a 
change  in  implementation  date  could  in  itself  modify  the  probabilities  of 
success.  Where  a number  of  interrelated  projects  are  being  planned,  some 
level  of  project  scheduling  may  be  required  using  PERT,  CPM,  etc.,  before 
reasonable  estimates  may  be  made. 

4.  For  each  of  the  above  estimates,  provide  an  expression  of  confidence  by 
estimating  in  each  case  an  optimistic  value  given  the  most  favorable 
conditions  occur,  and  a pessimistic  value  considering  the  most  adverse 
conditions  come  about. 

5.  Document  results  of  these  estimates  for  incorporation  into  the  project 
f i 1 e . 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

SUBTASK:  MAKE  PRELIMINARY  "RANGE-OF-EFFECT"  ESTIMATE 


DESCRIPTION 

If  the  process  of  describing  the  innovation  and  conducting  the  preliminary 
feasibility  analysis  has  not  provided  a clear  basis  for  a decision  regarding 
the  innovation,  this  subtask  will  provide  an  initial  quantitative  basis  for 
the  selection  of  a decision  route. 

PROCEDURE 

1.  Using  the  target  applications  for  the  innovation  from  Task  1;  i.e.,  course, 
jobs,  equipments,  students,  etc.,  make  a preliminary  estimate  of  the  ex- 
tended range  of  courses,  jobs,  etc.,  over  which  the  proposed  innovation 
might  apply. 

COMMENT:  If  data  bases  are  available  for  searching,  preliminary  search  para- 

meters may  be  defined  without  the  full  analysis  required  by  Task  5 (Determine 
Range-of-Effect) . 

2.  Identify  the  appropriate  cost  categories  impacted  by  the  innovation.  A 
further  subdivision  of  costs  between  development/implementation  and  re- 
curring operational  costs  should  be  maintained  so  that  costs  can  be  re- 
lated to  the  specific  outcomes  of  the  decision  tree.  For  example,  one 
class  of  costs  may  be  expended  in  the  process  of  attempting  to  implement, 
and  even  though  the  implementation  was  unsuccessful,  these  costs  will  be 
incurred  and  identified  in  outcome  C of  the  decision  tree  shown  in  Figure 
4.2.1.  Shutdown  costs  in  case  of  failure  should  be  considered. 

3.  Prepare  gross  cost/savings  estimates  for  each  cost  category  considering 
each  outcome  of  the  decision  tree  in  Figure  4.2.1. 

COMMENT:  If  a cost  model  is  available,  input  parameters  should  be  specified 

and  the  model  run  to  determine  preliminary  costs/savings. 

4.  .Enter  costs  on  the  decision  tree  for  outcomes  A through  D and  specify  the 

probabilities  (most  likely)  at  points  U through  Z.  An  example  of  a 
completed  decision  tree  is  shown  in  Figure  4.5.1. 

5.  Fold  back  the  decision  tree  to  obtain  the  preliminary  value  of  each  of 
the  three  decision  paths.  The  steps  in  the  foldback  process  are  illus- 
trated in  Figure  4.5.2. 
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6.  Select  preliminary  decision  path.  The  highest  positive  value  on  legs  1, 

2,  etc.,  would  be  an  indication  of  the  best  preliminary  decision.  If  all 
of  these  values  are  zero,  based  on  this  preliminary  assessment  the  proposed 
innovation  should  be  rejected.  It  is  possible,  however,  that  the  identifica- 
tion of  alternative  risk  reduction  projects  which  produce  revised  success 
probabilities  could  change  this  preliminary  result.  Also,  a revised  range- 
of-effect  analysis  may  expand  savings  beyond  cost  to  create  a more  favorable 
decision  outcome. 

7.  Document  results  of  decision  tree  analysis  for  incorporation  into  the 
project  file. 
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FIGURE  4.5.1.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
(EXAMPLE  OF  FORM  COMPLETED  WITH  INITIAL  OUTCOME 
VALUES  AND  PROBABILITIES) 
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FIGURE  4.5.2.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
(EXAMPLE  OF  FOLDING  BACK  PROCESS) 
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TASK:  PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT 

SUBTASK:  CREATE  TICKLER  FILE  ENTRY 


PROCEDURE 

1.  If  a decision  is  made  at  this  time  not  to  proceed,  the  problem  areas  should 
be  reanalyzed  to  establish  the  basis  upon  which  the  innovation  might  be 
reactivated.  For  example: 

• When  might  presently  insufficient  funding  be  available. 

• When  will  an  appropriate  level  of  technical  support  be  available. 

• When  might  existing  organizational  constraints  be  modified. 

t When  might  attitudinal  acceptance  on  the  part  of  the  user  population 
be  improved  to  bring  about  a more  acceptable  level  of  risk. 

• When  might  a particular  technological  breakthrough  occur. 

2.  Create  a tickler  file  entry  which  reflects  an  estimated  date  where  risk 
from  identified  problem  areas  may  be  reduced  so  that  the  proposed  innova- 
tion may  now  be  acceptable. 

RATIONALE 

One  of  the  objectives  of  the  ETAM  procedure  is  that  information  not  be  lost 
either  because  of  summarization  exercises  or  due  to  a gap  in  evaluation  or 
implementation  of  the  proposed  innovation.  Changing  conditions  may  create 
conditions  where  application  of  the  innovation  appears  more  reasonable.  It 
is  important  that  a system  of  estimating  the  time  at  which  these  conditions 
may  occur,  and  a system  of  forcing  visibility  of  the  proposal  at  that  time, 
be  instituted.  This  procedure,  while  seemingly  trivial,  is  intended  to  pre- 
vent loss  of  perhaps  valuable  data. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 

This  is  likely  to  be  the  most  important  and  difficult  task  in  the  entire 
procedure.  Its  general  purpose  is  to  determine  the  full  range  of  potential 
applicability  of  an  innovation  in  training,  operations,  and  in  the  personnel 
subsystem.  These  effects  may  consist  of  job  performance  and  training  ob- 
jectives for  which  "benefits"  are  yielded.  The  effects  may  also  be  potential 
savings  of  displaceable  dollars,  time,  level  of  talent  required  for  instruc- 
tional function,  or  other  cost  element.  The  latter  will  be  called  "costs  and 
cost  analysis".  Where  possible  a benefit  in  achieving  a training  objective 
(a  knowledge  or  skill)  will  be  translated  into  training  cost  savings. 

The  scope  of  range-of-effect  consists  of  many  factors,  concepts  and  operations. 
Simplifying  maps  may  be  misleading  if  taken  literally  or  as  complete  in  them- 
selves. Nevertheless,  the  reader  is  given  several  levels  of  abstraction  for 
Task  5: 

The  schematic  diagram  of  major  procedural  functions  that  is  presented 
at  the  head  of  this  chapter. 

The  listing  of  subtask  names  which  inmediately  follows  these  Introductory 
paragraphs. 

The  listing  of  subtasks  and  variables  and  categories  within  each  subtask: 
this  list  follows  the  subtask  name  listing. 

The  explanatory  descriptions  of  subtasks  and  categories  within  subtasks 
that  make  up  the  body  of  Task  5. 

An  Appendix  A which  treats  in  analytic  detail  some  of  the  key  characteriza- 
tions in  the  main  text  of  Task  5. 

The  reader  is  not  advised  to  use  the  contents  of  the  first  three  Items  above  as 
self-sufficient  either  for  critiquing  or  for  using  the  Range-of-Effect  procedures. 

The  quantitative  objectives  in  determining  range-of-effect  in  the  results  of 
training  can  be  expressed  in  student  training  hours  to  achieve  a training  ob- 
jective, minimum  levels  of  student  aptitude  requirements,  attrition  rate,  and 
level  of  student  performance.  These  variables  are  not  independent  of  each 
other,  so  assumptions  must  be  made  as  to  which  will  be  fixed,  and  which  will 
be  varied  for  assessment  purposes.  By  determining  the  range  of  jobs  and  tasks 
relevant  to  the  innovation,  the  range  of  courses  and  numbers  of  students  can 
be  identified. 

A second  class  of  quantitative  objective  is  the  determination  of  displaceable 
costs  which  the  innovation  might  achieve  in  the  Navy's  inventory  of  instruc- 
tional vehicles,  including  instructors,  or  to  displaceable  costs  in  training 
development  and  management  functions. 
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The  range-of-effect  procedures  consist  of  two  major  parts.  One  part  consists 
of  classificatory  terms  from  which  the  assessor  can  select  descriptors  which 
fit  the  innovation,  and  which  can  be  used  to  search  the  data  base  of  jobs  and 
job  tasks,  courses  and  course  objectives,  instructional  vehicles,  and  their 
properties.  The  results  of  the  data  base  searches,  when  checked  by  the  assessor, 
represent  the  full  pattern  of  students,  courses,  instructional  operations, 
vehicles  of  delivery,  and  other  elements  to  which  the  benefits  and  cost  advan- 
tages, if  any,  of  the  innovation  could  appropriately  apply. 

Note  that  the  "data  base"  may  exist  in  the  head  of  the  assessor  and  his  asso- 
ciates, on  paper  documents,  or  in  computerized  files  for  selective  access. 

Subject  matter  search  terms  are  never  completely  exhaustive  or  unambiguous. 

Human  judgment  finally  determines  whether  the  search  operation  has  indeed  made 
a useful  match  between  the  two  entities  described;  the  properties  of  the 
innovation  and  the  properties  of  the  entity  to  be  matched  with  it.  Even  though 
two  descriptions  may  match  each  other,  the  objects  denoted  by  the  search  may, 
when  examined  in  their  fuller  contexts,  have  important  differences  that  over- 
ride their  similarities.  The  user  should  be  aware  of  the  limitations  of  words 
in  describing  things. 

After  identifying  the  list  of  entities  to  which  the  innovation  has  potential 
relevance  (courses,  job  tasks,  aptitudes  or  instructional  vehicles,  or  train- 
ing development  and  administrative  functions),  the  second  major  activity  for 
the  assessor  is  to  assign  magnitudes  of  benefit  likely  to  be  conferred  by  the 
innovation.  He  also  indicates  the  likelihood  of  achieving  the  benefit  by  giv- 
ing an  expression  of  confidence  to  his  magnitude  estimate.  If  the  innovation 
offers  several  kinds  of  benefit,  he  rates  the  relative  importance  of  each. 

The  assessor  also  identifies  and  evaluates  secondary  benefits  to  be  expected 
of  the  innovation,  and  liabilities  which  its  adoption  might  bring  about. 

The  formats  on  which  the  assessor  develops  all  of  the  information  in  this  task 
are  intended  to  serve  as  procedural  aids  to  him.  They  also  transmit  the  data 
he  generates  into  the  cost  analysis  and  business  assessment  operations  that 
lead  to  an  assessment  outcome. 

Because  the  assessor  must  select  his  pathway  through  range-of-effect  analysis 
by  knowing  what  is  ahead  of  given  choice  points  of  route,  he  must  have  a good 
working  knowledge  of  the  entire  classificatory  scheme  in  this  task.  Arriving 
at  a description  of  the  application  potential  of  the  innovation  is  inductive 
and  insightful  as  well  as  deductive.  For  this  reason,  some  major  subtasks 
consist  of  activity  called  "orientation"  which  is  the  learning  of  information 
context  necessary  to  select  other  subtasks. 

SUBTASK  REFERENCE 

1.  Choose  the  major  route  for  determining  range-of-effect  05.01.01 

2.  Develop  background  for  instructional  vehicle  analysis  05.02.01 
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REFERENCE 

3. 

Select  vehicle  type  that  fits  the  innovation 

05.03.01 

4. 

Select  class  of  training  objective 

05.04.01 

5. 

Select  vehicle  properties 

05.05.01 

6. 

Complete  the  creating  of  specifications  for  searching 
the  instructional  vehicle  inventory 

05.06.01 

7. 

Identification  of  relevant  inventory  of  training  vehicles 

05.07.01 

8. 

Identify  benefit  pattern  for  instructional  vehicles 

05.08.01 

9. 

Develop  background  for  task  content  training  analysis 

05.09.01 

10. 

Identify  the  learning  objective 

05.10.01 

11. 

Make  preliminary  evaluation  of  benefit  importance 

05.11.01 

12. 

Determine  if  the  benefit  applies  to  skill  or  knowledge 
training 

05.12.01 

13. 

Select  task  structure  element  or  task  "function" 

05.13.01 

14. 

Identify  relevant  jobs  and  tasks 

05.14.01 

15. 

Select  relevant  stage  of  learning 

05.15.01 

16. 

Identify  relevant  courses 

05.16.01 

17. 

Identify  relevant  training  units  within  courses 

05.17.01 

18. 

Assess  the  training  benefit  pattern 

05.18.01 

19. 

Select  type  of  knowledge 

05.19.01 

20. 

Select  applicable  type  of  reference  knowledge 

05.20.01 

21. 

Identify  type  of  enabling  knowledge 

05.21.01 

22. 

Identify  type  of  task  format 

05.22.01 

23. 

Assess  on-the-job  benefits 

05.23.01 

24. 

Identify  relevant  function  in  development  and 
administrative  management  of  training 

05.24.01 
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TOPICAL  OUTLINE  OF  RANGE-OF-EFFECT 

This  listing  should  serve  as  a topical  guide  to  the  content  of  Task  5,  and 
after  knowledge  of  the  item  definitions  and  descriptions  in  the  main  text 
which  follows,  as  a kind  of  checklist.  The  subtasks  are  sequenced  by  logical 
order  rather  than  by  procedural  order  so  as  to  avoid  redundancies. 

05.01  Choose  the  major  route  for  determining  range  of  effect 

Instructional  vehicles 
Task  content  training 

Development  and  administrative  management  of  training 

INSTRUCTIONAL  VEHICLES 

05.03  Vehicle  types 

Instructor 
Static  graphics 
Animated  graphics 
Static  physical  models 
Procedural  trainers 

Symbol i c 
Physical  mockup 

Task  simulators 

05.04  Class  of  training  objective 
Reference  knowledge 

Knowledge  --  task  specific  and  enabling 
Task-skill  formats 
Skill  training 

05.05  Vehicle  properties 
Instructional  functions: 

Type  of  stimulus  presented  to  student 

visual 

auditory 

kinesthetic 

tactile 
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Type  of  content  displayed 

text-verbal 

diagrammatic 

representational 

abstracted  pictorial  representation 
pictorial  representations 
physical  representations 

Type  of  presentational  sequence 

library  of  frames  or  items 
presentation  sequence  not  applicable 
fixed  sequential  frames  or  items 
random  selection  of  frame  sequences 
dynamic  change  of  content  within  frame 

Selection  source  for  sequencing 

internal  program 
instructor 
student  choice 
student  performance 
combinations  of  the  above 

Type  of  external  control  operated  by  student 

not  applicable  directly 

artificial  or  symbolic  response  by  student 

representational  response  by  student 

by  symbolic  selection 
by  dummy  control  activation 

task-manipulative  response 

non-dynamic  in  time  and  force 
dynamic  and  interactive 

Feedback  presentation  logic 

not  applicable 

selects  next  stimulus  item  or  sequence 
gives  evaluation  of  preceding  response 
selects  and  presents  guidance  information 
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Response  evaluation  logic 

not  internal  — depends  on  Instructor  evaluation 
evaluation  limited  to  student's  inmedlate  response 
evaluation  extended  to  a set  of  student  responses 
tolerance  limits  on  acceptable  student  response 

fixed  , 

variable 

Utilization  functions: 

Programmability,  and  control  of  device  operation 
Portability,  including  storage 
Multiple  usage 

Set  up  and  operating  simplification 
Maintenance  simplification;  dependability 
Updatability  in  instructional  function 
Combining  instructional  functions 
Modularization  of  training  content 

Direct  translatablli ty  from  a requirement  to  training  decisions 
and  programming  of  training  content 

Supplemental  training  functions 

Extended  range  of  control  of  learning  conditions 
Extended  range  of  subject  matter  for  practice 
Extended  range  of  task  functions  trained 
Instructor  roles  facilitated 

TASK  CONTENT  TRAINING 

05.13  Task  structure  element  or  task  function 

Goal  projection 
Scan-detect 
Identify 
Interpret 

Procedure  following 
Decide 

Construct-plan 

Track 

Motor  performance 
Interpersonal  interaction 
Recall  task-cycle  Information 
Recall  enabling  information 
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05.14  Relevant  Jobs  and  Tasks 

Gross  job  categories 

Operations 

Maintenance 

Service 

Command 

Objective  task  variables 

Equipment  and  objects  used 
Environments  in  which  is  task  performed 
Tools  used 

Reference  Information  used  in  performing  task 
Task  operations  ("task  structure  elements") 

Criteria  of  task  performance 

05.15  Stage  of  Learning 

Orientation  and  familiarization 

Task  nomenclature  (identifications,  locations  and  names,  facts, 
rules) 

Task  formats  at  the  conceptual  level 
Procedures  learned  at  the  verbal  level  only 
Performing  task  components  with  guidance 

Doing  the  entire  task/ job  procedural ly:  barely  acceptable  mastery 

Highly  proficient  task/job  performance  in  work  context 

Unusual  task  conditions 

Performing  job/ tasks  at  key  man  level 

Refresher  learning 

05.19  Type  of  knowledge 

Reference 

Enabling 

05.20  Reference  knowledge 

System  purposes 
Organizational  roles 
Contexts  of  operation 
Organizational  rules  and  constraints 
Non-work  directed  content 

05.21  Enabling  knowledge 

Operational  goal  criteria 

Nomenclature,  identification  and  location  of  work  objects 
Procedure  descriptions 
Job  relevant  facts 
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05.22  Type  of  task  format 

Procedure  formats 
Decision  formats 
Construction  formats 


05  00  13 


05.18  Benefit  pattern  in  training:  variables 

Training  time  reduced 
Aptitude  level  reduced 
Attrition  level  reduced 
Performance  level  increased,  extended 

05.23  Benefit  pattern  on-the-job 

Greater  flexibility  in  assignment 

Less  on-the-job  training  to  acceptable  performance 

Higher  productivity  levels,  and  better  quality,  fewer  errors 

Greater  range  in  work  competence 

Willingness  under  stress 

Higher  level  of  self-initiated  and  self-directed  activity 
DEVELOPMENT  AND  ADMINISTRATIVE  MANAGEMENT  OF  TRAINING 


05.24  Development  and  Administrative  Management  of  Training 

Capture  and  description  of  skill  and  knowledge  requirements 
Matching  the  skill  requirement  with  descriptions  of  available 
resources  in  skills,  aptitudes  to  learn 

Matching  the  skill  requirements  with  available  training  resources 
Developing  training  requirements  from  skill  requirements 
descriptions 

Selecting  the  training  environments 

Slow  vs.  fast-paced  development  of  training  material 

Selection  or  devising  of  training  modules 

Selection  of  instructional  vehicles  and  media 

Designing  the  content  of  instruction 

Devising  selection  procedures  for  students 

Assessment  of  the  student 

Assignment  of  graduates  to  job  positions 

Evaluation  on  the  job 

Career  development 

Administrative  planning  and  control 

Evaluation  of  learning  or  entropy  of  the  training  system 
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SUBTASK:  CHOOSE  THE  MAJOR  ROUTE  FOR  DETERMINING  RANGE-OF-EFFECT 


There  are  three  major  routes  to  be  followed  for  the  assessment  of  the  innovation. 

Following  one  route  does  not  exclude  the  possibility  of  using  elements  in  another 

route.  The  alternatives  are: 

• Instructional  Vehicles.  The  basis  for  choosing  this  alternative  is  that 
the  innovation  aims  primarily  at  reducing  a cost  in  the  development  or  use 
of  an  instructional  vehicle  while  maintaining  a level  of  function  equivalent 
to  the  vehicle  (or  vehicle  property)  displaced  by  the  innovation.  The 
choice  of  this  route  leads  to  Subtask  05.02,  Develop  Background  for  Instruc- 
tional Vehicle  Analysis. 

• Task  Content  Training.  This  analytic  route  is  chosen  if  the  innovation  is 
directed  primarily  at  increasing  the  effectiveness  or  efficiency  of  in- 
structional processes  that  may  result  in  shorter  training  time  to  achieve 
a type  of  training  objective,  lower  aptitude  requirement,  lower  attrition 
rates,  or  a higher  level  of  learned  performance.  The  choice  of  this  route 
leads  to  Subtask  05.09,  Develop  Background  for  Task  Content  Training  Analysis. 

• Development  and  Administrative  Management  of  Training.  These  are  functions 
that  deal  with  the  information  interfaces  to  and  from  training,  such  as 
training  requirements  input  or  graduate  competence  descriptions  as  output. 
Others  deal  with  such  factors  as  space,  selection  of  instructional  media, 
development  of  training  course  content,  and  administrative  control.  At  the 
present  stage  of  formatting  these  processes,  an  innovation  dealing  with  any 
one  or  more  functions  must  be  restricted  to  showing  one  or  more  kinds  of 
cost  advantage  compared  with  the  reference  conditions  against  which  the 
innovation  is  to  be  evaluated.  This  choice  leads  to  Subtask  05.24,  Identify 
Relevant  Function  in  Development  and  Administrative  Management  of  Training. 

PROCEDURE 

1.  On  the  basis  of  applying  the  description  of  the  Innovation  prepared  in 
Task  1 to  the  considerations  above,  select  at  least  tentatively,  the 
route  for  assessment  of  potential  benefits  from  the  alternatives: 

Instructional  Vehicles 

Task  Content  Training 

Development  and  Administrative  Management  of  Training. 

2.  If  the  information  provided  in  Subtask  05.01  Is  Insufficient  for  making  this 
choice,  the  assessor  should  study  the  descriptive  and  analytic  content  in 
all  subtasks  in  Task  5,  and  then  make  his  choice  of  route  for  analysis. 
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SUBTASK:  DEVELOP  BACKGROUND  FOR  INSTRUCTIONAL  VEHICLE  ANALYSIS 


DESCRIPTION 

Instructional  vehicles  range  from  a printed  page  of  instructions  through  human 
instructors  and  to  the  actual  work  objects  and  tools  --  combined  with  the 
learner  and  the  work  environment.  This  range  defies  any  simple  classification 
of  instructional  vehicle  types,  purposes,  functions,  or  characteristics. 

Determining  range-of-effect  of  an  innovation  through  the  route  of  instructional 
vehicle  characteristics  should  be  based  primarily  on  the  following  condition: 
the  innovation  is  intended  to  reduce  or  displace  existing  or  expected  costs  on 
an  existing  or  planned  inventory  of  instructional  vehicles.  The  cost  reduction 
assumes  no  loss  in  training  benefit,  and  is  not  to  be  evaluated  primarily  on  an 
increase  In  the  training  benefit.  Thus,  the  substitution  of  digital  computers 
for  analog  computers  in  complex  simulators  is  likely  to  be  based  primarily  on 
cost  comparisons  and  cost  generalizations  rather  than  on  training  outcome 
benefits.  When  the  innovation  is  directed  primarily  to  an  Increased  training 
benefit,  the  assessor  should  generally  choose  the  Task  Content  Training  Route 
(05.09)  for  determining  the  innovation's  range-of-effect. 

The  purpose  to  be  served  by  a classiflcatory  system  for  Instructional  vehicles 
Is  to  provide  a set  of  descriptors  for  linking  the  innovation  to  the  appropriate 
subset  of  training  vehicles  in  the  entire  Navy  inventory  of  training  vehicles. 
Thus,  if  a grossly  cheaper  way  of  programming  dynamic  three-dimensional  repre- 
sentations on  two-dimensional  displays  is  the  proposed  innovation,  one  wants  to 
Identify  all  items  in  the  inventory  that  have  visual  displays  that  present  or 
simulate  three-dimensional  motion.  Potentially  relevant  vehicles  may  not  be 
restricted  to  complex  training  simulators;  they  could  also  apply  to  knowledge 
trainers  that  instruct  in  principles  of  mechanical  motion,  or  to  perceptual 
trainers  for  the  scanning,  detection,  and  Identification  of  terrain  Information. 

The  assessor  has  two  major  routes  for  identifying  the  relevant  training  vehicle 
inventory.  He  should  .look  ahead  beyond  the  choice  point  and  see  if  any  descrip- 
tors in  either  of  the  routes  make  a good  first  order  fit.  If  so,  that  is  the 
direction  he  should  take,  at  least  at  the  start. 

PROCEDURE 

1.  Become  familiar  with  the  conditions  for  selecting  the  instructional 
vehicles  route  for  determining  the  range-of-effect  for  the  Innovation. 

2.  Read  the  description  of  the  innovation  given  in  Task  1. 

3.  Examine  the  terms  in  the  listings  under  Subtasks  05.03,  Select  Vehicle 
Type,  and  05.05,  Select  Vehicle  Properties.  Some  one  or  combination  of 
these  terms  may  suggest  a direct  way  of  describing  the  innovation  for 
searching  the  Instructional  vehicle  inventory. 
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4.  Do  not  be  limited  to  the  description  of  the  vehicle  by  any  terminology, 
however.  If  there  are  other  terms  which  can  more  quickly  and  compre- 
hensively identify  the  inventory  items,  use  them. 

5.  If  the  Innovation  is,  without  doubt,  limited  to  an  already  identified 
piece  of  equipment,  or  a medium  used  In  a given  way,  It  is  unnecessary 
to  go  through  an  elaborate  search  operation.  Go  directly  to  Sub task 
05.05,  Select  Vehicle  Properties.  It  is  possible,  however,  that  the 
innovation,  either  as  a device  or  as  a construction  principle,  may  be 
applicable  to  a class  of  vehicles  with  which  the  assessor  is  not  familiar. 
The  data  base  search  on  selected  descriptors  would  then  be  a safe  bet. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  VEHICLE  TYPE  THAT  FITS  THE  INNOVATION 


DESCRIPTION 

A vehicle  Is  anything  that  performs  instructional  functions;  for  purposes  of  the 
present  analysis,  this  Includes  the  Instructor.  The  Innovation  may  apply  more 
of  less  equally  to  all  members  of  a given  type  of  instructional  vehicle;  for 
example,  Interpersonal  skills  in  instructors,  or  a device  for  creating,  storing, 
and  Indexing  microfiche  and  other  images.  The  fastest  means  of  locating  the 
relevant  inventory  Is  to  use  the  broadest  name  that  covers  the  range,  subsetting 
with  qualifying  terms  if  the  preliminary  results  of  the  data  base  search  yield 
too  many  irrelevant  hits. 

The  names  In  the  list  of  vehicle  types  cited  below  should  be  understood  in  terms 
of  the  definitions,  cited  in  the  text  under  PROCEDURE.  Each  of  the  terms  cited 
may  have  somewhat  different  meanings  outside  of  the  present  context.  Many  of 
the  distinctions  made  here  among  the  types  are  primarily  derived  from  the  set  of 
"Instructional  Functions"  listed  under  Subtask  05.05,  Select  Vehicle  Properties. 

The  major  vehicle  types  consist  of: 

Instructor 
Static  graphics 
Animated  graphics 
Static  physical  models 
Procedural  trainers 

Symbolic 
Physical  mockup 

Task  simulators. 

PROCEDURE 

1.  If  the  innovation  has  to  do  with  all  members,  or  a large  subset  of  members, 
of  any  one  of  the  following  types  of  Instructional  vehicles,  select  the 
term  to  which  the  innovation  applies. 

• The  Instructor.  Innovations  may  be  directed  towards  improvements  in  the 
selection,  training,  assignment,  motivation,  and  turnover  of  instructors. 
The  "Improvement"  should  be  assessable  In  terms  of  reductions  of  the  norms 
against  which  the  innovation  Is  targeted.  Less  training  time  to  achieve 
a given  instructional  competence  is  directly  measurable.  An  Innovation 
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that  enables,  with  the  normal  training  time  and  cost,  an  instructor  to 
teach  a greater  variety  of  courses  must  be  more  indirectly  assessed  in 
terms  of  probable  dollar  value  of  the  capability  in  the  expected  train- 
ing environment.  If  instructor  versatility  is  expected  to  reduce  in- 
structor turnover  rates,  the  expected  reduction  of  rate  (and  its 
probability)  will  have  to  balance  against  the  tangible  values  in  re- 
ducing that  rate. 

The  assessment  picture  may  include  estimates  of  intangible  factors  such 
as  "Instructor  satisfaction"  or  "instructor  Interest  in  what  he  is 
teaching",  but  should  be  Identified  as  Intangibles  — not  because  that 
makes  them  unimportant,  but  to  identify  their  intangibility. 

The  instructor  is,  or  should  be,  the  key  instructional  vehicle  in  the 
entire  training  establishment.  He,  therefore,  justifies  large  efforts 
to  make  him  effective  in  his  varied  teacher  roles.  But  he  may  also  be 
the  target  of  many  well-meaning,  poorly  worked- through  and  inadequately 
tested  innovations  which,  had  they  been  assessed  on  tough-minded 
criteria,  would  quickly  have  been  disregarded.  It  Is  because  the  in- 
structor is  vulnerable  to  many  "innovative"  influences,  most  of  which 
may  be  distractors  to  him,  that  assessment  screenings  can  be  useful  to 
him  as  well  as  to  the  establishment. 

The  Instructor  presents  Information  by  words,  chalkboard  diagrams, 
gestures,  and  sometimes  by  demonstrating  task  performance  literally  or 
symbolically.  In  the  classroom  he  is  effective  primarily  in  teaching 
concepts,  principles,  strategies,  and  aspiration  goals.  He  is  com- 
paratively inefficient  in  drill  on  material  learned  by  repetitive 
practice  (rote),  such  as  facts,  rules,  nomenclature,  perceptual  dis- 
criminations, and  classifications.  When  teaching  task  skills  he  may 
have  limited  Instructional  value  as  a model  to  copy  or  emulate  in  per- 
forming the  task.  He  may  have  great  value  in  responding  to  individual 
differences  among  students  (if  he  has  this  sensitivity)  and  tailoring 
aspirations  and  practice  content  for  the  student.  He  may  focus  or 
expand  student  attention  while  engaged  in  task  practice,  provide 
supplementary  task  cues,  and  supplement  normal  task  feedback  with  inter- 
pretive feedback  — relating  specifics  of  what  the  student  did  or 
failed  to  do  that  resulted  in  a good  or  a poor  outcome. 

Both  his  efficiency  and  effectiveness  as  an  instructor  depend  on  the 
goodness  of  his  implicit  process  model  of  the  task,  his  awareness  of 
the  kinds  and  stages  of  learning,  and  knowledge  of  the  Individual 
learning  characteristics  of  the  student.1  The  instructor  can  serve  as 
a human  model  In  the  establishment  of  positive  attitudes. 

• Static  Graphics.  A static  graphic  is  a display  on  a page  or  screen  of 
a content  that  does  not  move.  Text,  pictures,  diagrams,  maps,  and 
lists  fall  into  this  category.  Books,  charts,  foils  and  overlays,  slide 
projectors,  and  microfiche,  are  devices  in  this  category. 
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The  training  objectives  for  these  devices  tend  to  be  limited  to  the 
acquisition  of  reference  knowledge  and  in  some  cases  to  enabling 
knowledge.  The  latter  is  true  where  the  target  skill  is  symbolic, 
such  as  interpreting  the  symbols  on  a terrain  map,  but  in  these  cases 
there  must  be  supplemental  instructional  functions  brought  to  bear. 

Developmental  and  production  cost  for  graphic  aids  is  notoriously  high 
at  ordinary  commercial  rates.  For  example,  an  invention  that  would 
enable  individuals,  after  a few  minutes  of  training,  to  create  the 
content  and  control  the  process  for  generating  an  ordinary  colored 
slide  would  have  an  average  $30  to  $40  commercial  cost  per  slide  as 
a displaceable  target. 

Other  cost  factors  on  static  graphics  include  storage  and  retrieval. 

An  instructor  may  often  prefer  to  make  a new  graphic  rather  than  spend 
the  time  and  effort  required  to  find  an  existing  one  that  might  have 
served  the  purpose. 

• Animated  Graphics.  These  are  graphics  that  present  the  appearance  of 
motion  in  the  content  displayed.  Television,  the  motion  picture  and 
the  computer-generated  display  are  examples.  (There  is  an  important 
sense  in  which  the  image  being  drawn  by  the  instructor  on  a blackboard 
is  an  "animated  graphic",  although  its  completion  makes  it  a static 
graphic. ) 

Animated  graphics  may  transmit  concepts  of  serial  dependencies,  process 
sequences,  and  time  dependencies.  As  such,  they  may  contribute  to 
reference  knowledge.  They  may  be  task-specific  and  enabling  if  they 
provide  a mnemonic  structure  for  the  later  learning  of  task  procedures, 
or  for  the  generalizing  of  task  procedures  to  a variety  of  work  situa- 
tions not  immediately  apparent  to  the  student.  In  order  to  be  useful, 
the  mnemonic  structure  must  parallel  the  structure  of  what  is  to  be 
learned  as  a task  skill . 

Animated  graphics  may  be  pictorial  or  diagrammatic.  Both  are  commer- 
cially expensive  to  create,  the  diagrammatic  ones  far  more  so  than  the 
pictorial.  Computers  have  been  programmed  to  develop  graphic  sequences 
on  a display  tube  that  may  be  less  expensive  than  sequences  developed 
manually.  The  Walt  Disney  studios  have  an  extensive  technology  for 
creating  animation  effects. 

• Static  Physical  Models.  A physical  model  is  a three-dimensional  rep- 
resentation of  objects.  Such  representations  are  often  called  mock- 
ups.  Their  demonstration  effect  as  a value  to  training  may  be 

Questioned,  although  they  may  offer  a kind  of  reference  knowledge.  A 
emonstration  model  of  a variety  of  fighter  airplane  cockpits  may 
present  similarities  and  differences  among  them.  Three-dimensional 
representations  need  not  be  life  size. 
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A physical  model  may  be  used  to  teach  a "task-specific  enabling  know- 
ledge" by  providing  Instruction  on  nomenclature  and  locations  of  work 
objects. 

These  devices  are  generally  expensive  to  build,  costly  to  store,  and 
used  only  briefly  in  the  student's  learning  scenario. 

• Procedural  Trainers.  Procedural  trainers  not  only  present  the  student 
with  a display  of  information,  but  enable  him  to  make  (1)  actual  or 
(2)  simulated  task  responses  with  the  device  with  a feedback  repre- 
sentation. Procedural  trainers  may  provide  direct  practice  in  a 
cognitive  skill.  An  example  would  be  training  in  the  cognitive 
aspects  of  troubleshooting,  or  in  the  planning  of  a military  action. 

The  procedural  trainer  may  instruct  in  the  symbolic  representation  of 
a nontracking  perceptual -motor  task,  such  as  the  verbal  sequence  that 
describes  the  actions  to  take  in  shutting  down  a particular  type  of 
power  plant  in  a vessel.  The  latter  would  be  an  example  of  instruc- 
tion in  a task-skill  format  as  represented  by  the  student's  ability 
to  give  a verbal  description  of  the  procedure,  or  to  simulate  the 
steps  in  the  procedure  in  the  physical  task  environment. 

Procedure  trainers  may  be  physical  representations  of  the  task  en- 
vironment. The  controls  operated  by  the  student  may  simulate  actual 
functions,  or  act  merely  as  dummy  controls,  or  have  a variety  of 
feedback  cues  that  range  between  these  extremes.  There  is  some 
growing  evidence,  and  psychological  theory,  which  shows  that  the 
physical  fidelity  of  control  response  in  the  learning  and  generaliza- 
tion of  most  procedures  may  be  very  low,  at  least  through  the  early 
and  middle  levels  of  task  learning  where  extensive  practice  is  re- 
quired for  "learning  what  to  do  next"  with  a high  degree  of  reliability. 

Where  performance  data  are  collected,  the  large  percentage  of  human 
errors  are  always  assigned  to  "failure  to  follow  standard  operating 
procedure",  these  findings  imply,  among  other  things,  inadequate 
practice  by  students  on  procedures  as  such,  and  inadequate  practice 
of  procedures  in  full  work  context  if  the  latter  may  interfere  with 
recall. 

The  kinds  and  amounts  of  physical  fidelity  that  are  necessary  and 
sufficient  for  various  classes  of  procedure  training  are  poorly  under- 
stood, and  practices  used  in  designing  training  vehicles  for  procedure 
training  is  largely  ad  hoc.  But  because  the  costs  of  physical  fidelity 
are  high,  and  because  students  should  have  massive  amounts  of  practice 
in  order  to  acquire  reliable  procedural  skills,  the  opportunities  for 
innovations  leading  to  large  cost  reductions  in  overall  training  are 
perhaps  greater  in  this  field  than  any  other  except  in  the  determina- 
tion of  what  is  relevant  to  teach  and  learn. 

Evaluations  of  innovations  in  procedural  trainers  should  take  into 
account  the  Stages  of  Learning  analysis  (Subtask  05.15). 
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Factors  applicable  to  most  procedural  trainers  are  identified  under 
Select  Vehicle  Properties  (Subtask  05.05),  which  may  be  used  to  further 
refine  the  relevant  description  of  the  innovation. 

• Simulators  of  Operational  Devices.  These  devices  have  "realistic" 
physical  or  psychological  fidelity  to  the  work  task  situation.  The 
controls  used  by  the  operator,  and  control  feedback  and  other  features, 
are  intended  to  make  the  student  respond  as  if  he  were  performing  in 
the  real-life  situation.  The  device  may  also  contain  a variety  of 
sensors  and  interpreters  of  student  response  for  evaluation  and  control 
of  exercise  content  presented  to  the  student. 

Because  simulators  are  expensive  devices  to  develop,  program  and 
operate,  there  are  many  opportunities  for  displaceable  dollars.  The 
relationship  of  kind  and  degree  of  fidelity  of  physical  simulation  to 
transfer  of  training  is  a key  cost  factor.  Fidelity  of  stimulus  input 
Is  an  especially  sensitive  cost  which  is  understood  abstractly,  but  in 
the  specific  case  must  generally  be  solved  by  ad  hoc  inquiry  and  ex- 
pensive experimentation. 

Unfortunately,  the  high  cost  of  simulators  tends  to  restrict  the  number 
of  practice  hours  it  is  feasible  to  give  individual  students.  Some 
attention  has  been  directed  towards  the  use  of  less  costly  "part  task 
trainers"  to  achieve  part  skills  and  lower  learning  levels  of  skills. 

A major  Innovation  would  consist  of  a systematic  set  of  practical  guide- 
lines that  would  specify  criteria  of  minimum  physical  simulation  require- 
ments for  given  classes  of  tasks,  stages  of  learning,  and  degree  of 
task  Integration.  The  formulation  of  such  guidelines  would  necessarily 
rest  on  a theory  of  transfer  of  training  at  varying  levels  of  learning 
in  various  perceptual,  cognitive,  and  perceptual -motor  tasks. 

Simulators  are  also  used  to  evaluate  and  check  out  the  competence  of 
students  who  are  at  or  near  operational  levels  of  performance.  It  is 
unclear  whether  a simulator  that  is  used  for  evaluation  requires  any 
properties  that  differ  from  those  needed  for  high  level  training.  It 
is  assumed  that  the  purpose  of  the  evaluation  here  is  that  of  qualify- 
ing or  disqualifying  the  student  from  readiness  to  perform  on  opera- 
tional equipment. 

The  assessment  of  innovations  in  this  class  of  device  should  take  real- 
istic account  of  the  effect  of  training  management,  instructors,  and 
students  on  the  training  effectiveness  of  the  device.  It  is  well  known 
that  if  an  Instructor  dislikes  the  training  device  he  is  required  to 
use,  the  student  will  also  tend  to  dislike  or  distrust  it,  with  a 
corresponding  inadequacy  in  its  potency  for  learning.  This  could  mean 
that  a productive  innovation  might  consist  of  a means  of  inducing 
favorable  attitudes  towards  effective  training  devices  that  may  fail 
to  evoke  favorable  attitudes  from  teacher  and  student. 
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All  of  the  vehicle  properties  listed  in  Suhtask  05.05  apply  to  simula- 
tors. After  identifying  that  the  innovation  applies  rough.ly  to  sim- 
ulators, use  the  vehicle  properties  listings  to  further  refine  the 
characterization. 

2.  If  the  vehicle  type  appropriate  to  the  innovation  does  not  appear  by  name 
in  the  preceding  list,  search  Instructional  Functions  in  Subtask  05.05. 

For  example,  a trainer  to  identify  and  interpret  auditory  signals  would 
not  be  represented  in  the  above  list.  Trainers  in  the  vehicle  inventory 
with  this  property  would  be  identified  by  searching  under  05.05,  Instruc- 
tional Functions:  type  of  stimulus:  auditory. 

3.  Assemble  the  descriptor  names  that  fit  the  innovation  and  retain  them  for 
use  in  searching  the  data  base  of  the  instructional  vehicle  In  order  to 
determine  those  items  in  the  inventory  that  are  appropriate  to  the  innova- 
tion. 

4.  If  the  Innovation  deals  with  a type  of  vehicle  but  Is  restricted  to  a class 
of  training  objective,  a two-stage  description  and  search  will  have  to  be 
made.  The  second  stage  that  restricts  the  innovation  to  a given  set  of 
training  objectives  will  be  treated  In  Subtask  05.04,  Select  Class  of  Train- 
ing Objective. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  CLASS  OF  TRAINING  OBJECTIVE 


DESCRIPTION 

It  is  desirable  to  determine  the  widest  potential  applicability  of  the  innova- 
tion to  instructional  vehicles  because  the  economies  of  scale  may  transform  a 
minor  contribution  to  one  of  major  Importance.  But  the  application  of  the 
innovation  may  have  to  be  qualified  within  a vehicle  type  by  several  factors. 

It  may  be  restricted  to  a given  class  of  training  objective,  although  exten- 
sive to  all  examples  within  that  objective.  The  Innovation  may  also  be  limited 
to  a class  of  instructional  environment;  thus  slides  require  a slide  projector 
and  this  may  Imply  a space  requirement  that  limits  its  range  of  application. 

The  types  of  training  objectives  listed  below  are  general,  and  intended  only 
to  suggest  a training  context.  This  level  of  descriptors,  when  added  to  the 
vehicle  type  descriptor,  may  be  adequate  to  search  the  instructional  vehicle 
inventory  list  and  yield  a reasonable  range  of  candidates  for  manual  culling. 

PROCEDURE 

1.  Select  the  one  or  more  descriptors  which,  by  exclusion  of  the  other  de- 
scriptors in  the  set,  properly  restricts  the  range  of  applicability  of 
the  innovation. 

• Reference  Knowledge  (see  definition  in  Subtask  05.20). 

• Knowledge-Task  Specific  and  Enabling  (see  definition  in  Subtask  05.21). 

• Task-Skill  Formats  (for  definition  see  Subtask  05.22). 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  VEHICLE  PROPERTIES 


DESCRIPTION 

Assume  that  the  choice  has  been  made  to  identify  the  applicable  instructional 
vehicles  In  the  vehicle  inventory  by  the  specification  of  instructional  function 
or  utilization  function.  Presumably,  the  innovation  represents  a functional 
capability  that  is  fairly  independent  of  the  kind  of  training  content  which  the 
vehicle  Instructs. 

The  following  list  of  Instructional  Functions  is  not  arbitrary.  It  is  a fairly 
complete  and  definitive  set  of  functions  for  control  of  the  teaching  and  learn- 
ing processes.  Some  instructional  vehicles  may  in  themselves  contain  only 
several  of  these  functions.  A graphic  aid  such  as  a chart,  in  itself,  is  char- 
acterized only  by  a visual  stimulus  and  a given  type  of  content  such  as  a "flow 
diagram".  In  this  case,  the  remaining  functions  in  the  list  of  instructional 
functions  must  be  performed  manually  by  teacher  and  student. 

The  second  list.  Utilization  Functions,  cannot  claim  equivalent  comprehensive- 
ness, and  might  potentially  have  to  be  supplemented.  Almost  certainly  a Utiliza- 
tion Function  would  have  to  be  combined  with  other  descriptors,  such  as  Instruc- 
tional Function  or  type  of  vehicle  (Subtask  05.03),  in  developing  a usable 
search  argument  to  apply  to  a vehicle  inventory  listing. 

PROCEDURE 

1.  After  examining  the  description  of  the  innovation,  select  one  or  more  of 
the  instructional  functions  listed  below  to  which  the  innovation  applies. 

Set  down  both  the  name  of  the  function  (such  as  "Type  of  stimulus  presented 
to  the  student")  and  the  item  or  characteristic  listed  under  that  function 
which  applies  to  the  innovation  (such  as,  for  example,  "auditory").  If  no 
item  in  this  list  applies,  examine  the  list  of  Utilization  Functions  in 
Step  2. 

INSTRUCTIONAL  FUNCTIONS 

• Type  of  stimulus  presented  to  student 

Visual  — for  example,  the  display  of  simulated  landing  strip;  or  a 
flowchart  diagram;  or  printed  text. 

Auditory  — examples,  simulated  sonar  data;  sounds  of  defect  in  moving 
mechanical  assembly;  voice  transmission  through  noisy  channel. 

Kinesthetic  — examples,  control  force  in  aircraft  stick  and  rudder; 
torque  applied  to  a bolt;  the  feel  of  a potentially  broken  human  limb. 
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Tactile  — examples,  smoothness  of  surface  of  a mechanical  bearing; 
conducted  temperature  to  junctions  adjacent  to  one  being  soldered; 
texture  of  a type  of  cloth. 

• Type  of  content  displayed 

Text  - verbal  — examples:  continuous  text;  formatted  text  as  in  job 

instructions;  text  in  tabular  format. 

Diagrammatic  — examples:  flowcharts;  PERT  diagrams;  histograms;  logic 

charts;  decision  tables;  schematics. 

Representati onal 

Comment:  Representation,  as  applicable  to  instruction,  covers  a wide 

range  of  kind  and  degree.  The  topic  is  often  in  the  context  of  simula- 
tion fidelity  where  transfer  of  training  value  and  costs  have  complex 
Interaction  and  tradeoffs.  As  defined  here,  a "representational"  dis- 
play is  one  which  has  some  degree  of  isomorphic  identity  with  the 
physical  appearance  of  real-life  task  objects  and  situations.  The 
classification  here,  as  elsewhere  in  this  section,  is  based  on  what  the 
student  sees,  not  on  how  the  material  displayed  to  him  is  generated. 

The  following  three  items  roughly  cover  the  range  of  representational 
formats. 

Abstracted  pictorial  representation:  example  — a landing  runway 

displayed  only  as  a rectangle  in  changing  perspective  and  size  for 
training  in  aircraft  landings.  Secondary  cues  such  as  terrain, 
buildings  or  other  structures,  are  not  presented,  but  the  shapes 
of  the  rectangle  correctly  simulate  the  appearance  of  runways  under 
different  landing  conditions.  In  other  training  contexts,  the 
silhouette  of  aircraft  types  or  types  of  ships  would  be  abstracted 
pictorial  representations. 

Pictorial  representations:  images  of  actual  appearances  of  objects 

and  situations  such  as  projected  by  motion  picture  cameras,  photo- 
graphs or  realistic  drawings. 

Physical  representations:  examples,  the  actual  work  object  itself, 

or  three-dimensional  mockups. 

• Type  of  presentational  sequence 

Library  of  "frames"  or  items  — the  creation  of  a library  serving  as  a 
source  of  instructional  material,  but  where  the  items  as  stored  In  the 
library  have  no  inherent  sequentiality. 

Presentation  sequence  not  applicable  to  vehicle  — as  for  example,  the 
physical  model  of  a device,  or  a cutaway  model. 
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Fixed  sequential  frames  or  items  — for  example,  the  standard  Kodak 
carousel,  or  slide  film,  or  a continuous  motion  picture. 

Random  selection  of  frame  sequences  — sequence  may  be  conditional 
on  choice  of  user  or  program,  or  based  on  student  response  such  as 
in  branching  type  of  computer  assisted  instruction. 

Dynamic  change  of  content  within  frame  by  display  attributes  or 
variables  — as  is  typical  of  computer  controlled  simulators  where 
altitude  indication  may  change  independent  of  airspeed  indication, 
or  the  elements  in  a display  of  a simulated  display  may  remain  con- 
stant but  change  with  respect  to  simulated  point  of  regard. 

• Selection  source  for  sequencing 

Internal  program  — for  example,  a computer  program,  the  operation  of 
which  selects  a next  frame  or  item  where  the  selection  is  based  on 
student  response  to  a problem  presentation;  or  a fixed  mechanism  for 
advancing  each  next  frame  in  fixed  sequence. 

Instructor  --  selects  what  will  next  be  presented  to  the  student 
either  as  a specific  item  or  a sequence  of  related  items. 

Student  --  selects  next  item  of  presentation  by  voluntary  choice. 

Student  performance  — the  outcome  of  the  student's  response  to  a 
learning  item  determines  what  next  item  will  be  presented  to  him. 

Combinations  of  the  above  four  items. 

• Type  of  external  control  operated  by  student  (relevant  to  task  operations) 

Not  applicable  directly  --  for  example,  because  the  instructional  func- 
tion of  interest  is  not  linked  to  a student's  response. 

Artificial  or  symbolic  response  by  student  — such  as  when  the  student 
selects  from  multiple  choice  options  presented  verbally. 

Representational  (semi realistic)  response  by  student: 

By  symbolic  selection  — as  by  pictorial  display  of  control  switches 
keyed  to  an  answer  sheet. 

By  dumny  control  activation  — physical  switches  and  other  controls 
manipulated  by  student,  but  which  are  not  operational. 

Task-manipulative  response: 

Nondynamic  in  time  and  force  with  respect  to  changes  that  take  place 
in  the  presentational  display  of  the  task. 
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Dynamic  and  interactive  fidelity  (to  some  degree)  in  time  and 
control  forces  with  display  change. 

• Feedback  presentation  logic 

Not  applicable  to  the  Issue  being  investigated. 

Selects  next  stimulus  item  or  sequence  — what  Is  next  presented  to 
the  student  depends  upon  an  evaluation  of  the  student's  response. 

Gives  evaluation  of  preceding  response  to  student  --  the  Instructional 
vehicle  presents  the  student  with  information  that  to  some  degree 
critiques  his  response  or  set  of  responses.  This  may  range  from 
"right"  or  "wrong"  to  an  explanation  of  why  the  student's  response 
was  right  or  wrong  or  inadequate. 

Selects  and  presents  guidance  information  as  well  as  problem  data  -- 
the  guidance  data  presented  is  conditional  upon  the  student's  previous 
responses. 

• Response  evaluation  logic 

Not  internal  --  depends  on  instructor  evaluation  of  student's  response, 
or  the  student's  evaluation  of  his  response. 

Evaluation  limited  to  student's  immediate  response. 

Evaluation  extended  to  a set  of  student  response,  and/or  to  classes 
of  student  response  — for  example,  as  in  a debriefing  of  a simulated 
mission  where  the  student's  general  strategy  is  evaluated. 

Tolerance  limits  on  acceptable  student  response: 

Fixed  --  in  terms  of  acceptable  or  non-acceptable  response. 

Variable  — such  as  dependent  on  amount  of  practice  or  stage  of 
learning  the  task. 

More  than  one  of  the  foregoing  instructional  functions  may  apply  to  an 
innovation.  Furthermore,  the  innovation  may  not  necessarily  apply  to  a 
mechanized  function,  but  to  human  activities  In  the  Instruction/learning 
process. 

It 

The  reader  is  reminded  that  the  concern  in  the  Instructional  Vehicles 
section  of  determining  range  of  effect  of  the  innovation  is  directed  towards 
displaceable  costs  by  substituting  the  innovation  for  an  existing  or  pro- 
jected alternative.  Where  the  training  function  is  intended  to  have  a 
benefit  in  what  is  learned  by  the  student,  the  assessment  should  include 
the  analytic  route  called  "Task  Content  Training  Analysis".  It  is  this 
distinction  which  rules  out  such  factors  as  student  motivation,  student 
aptitudes,  learning  efficiency  from  consideration  in  this  Instructional 
Vehicles  section. 
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2.  Select  one  or  more  applicable  Utilization  Functions  from  the  following  list. 
Note  that  whereas  innovations  may  improve  the  "convenience"  of  any  of  these 
functions  in  a device,  It  is  generally  possible  to  translate  convenience 
into  some  kind  of  dollar  value  as  a justification. 

UTILIZATION  FUNCTIONS 

• Progranmabillty  and  control  of  device  operation. 

• Portability,  including  storage. 

• Multiple  usage  --  as  of  job  aid  as  well  as  training  vehicle. 

• Set  up  and  operating  simplification  — requiring  less  ability  and  train- 
ing time  for  the  instructor  to  use  effectively,  or  that  enables  the 
student  to  perform  the  operations. 

• Maintenance  simplification;  dependability. 

• Updatability  In  instructional  function. 

• Combining  instructional  capabilities  heretofore  distributed  among  several 
medi a/devices. 

• Modularization  of  training  content  for  more  general  applicability. 

• Direct  translatabi 11 ty  from  a format  for  describing  job  task  require- 
ments into  training  decisions  and  the  selection  or  programming  of  train- 
ing content  and  sequence.  (This  item  is  also  treated  in  Subtask  05.24.) 

Note  that  a Utilization  Function  in  itself  cannot  be  effectively  used  to 
search  a data  base  of  instructional  vehicles.  It  will  have  to  be  combined 
with  additional  classificatory  terms  such  as  Vehicle  Type  and,  very  likely, 
subsets  within  Vehicle  Type  beyond  the  descriptors  in  the  range-of-effect 
section.  The  device  context  of  the  Innovation  itself  may  provide  more  or 
less  self-evident  characterizations  to  be  added  to  the  search  profile,  or 
may  point  directly  to  the  set  of  relevant  vehicles  to  an  individual  with  a 
general  knowledge  of  the  Navy's  Inventory  of  instructional  vehicles.  (For 
an  exhaustive  listing  of  media  and  Instructional  vehicles,  see  the  report 
by  Training  Analysis  and  Evaluation  Group  (TAEG),  A Technique  for  Choosing 
Cost-Effective  Instructional  Media,  Appendix  B;  published  April  1974. 

3.  Identify  training  functions  that  may  be  supplementary  benefits  to  dis- 
placeable costs.  These  supplementary  benefits  will  be  useful  in  formulating 
the  benefits  picture  for  the  Innovation  beyond  the  primary  benefit  of  cost 
advantage.  It  should  be  understood,  however,  that  a formal  assessment  of 
the  innovation's  value  for  an  increased  training  benefit  in  generating 
knowledge  and  skill  in  students  must  go  through  task  content  training  assess- 
ment as  in  Subtask  05.09. 
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TRAINING  FUNCTIONS  TREATED  AS  SUPPLEMENTAL  BENEFITS  TO  DISPLACEABLE  COSTS 

• Extended  Range  of  Control  of  Learning  Conditions.  Taking  learning  to 
a higher  level  of  performance  or  expertise  on  a task  or  task-cluster. 
May  be  by  enabling  more  practice  on  the  same  materials  to  become 
practicable,  or  by  modifying  presentation  content  and  units  and  feed- 
back, or  modifying  rate  of  action  demand,  or  by  fostering  the  integra- 
tion of  part- tasks  already  learned. 

• Extended  Range  of  Subject  Matter  for  Practice.  This  can  enable  a skill 
to  be  generalized  to  the  range  of  conditions  encountered  in  real  life, 
coping  with  contingencies,  and  adapting  to  environmental  constraints 
and  opportunities. 

• Extended  Range  of  Task  Functions  on  Which  Training  is  Given.  For 
example,  a device  specialized  for  signal  detection  and  identification 
becomes  adapted  to  instruction  on  Interpretation  of  the  meaning  of 
information  patterns,  or  to  coping  procedures  and  strategies. 

• Instructor  Roles  Facilitated. 

a.  The  instructor's  capabilities  may  be  extended  for  the  purpose  of 
sharpening  instruction  to  the  individual  or  group. 

b.  Clerical  requirements  are  reduced  so  that  greater  time  and  atten- 
tion can  be  given  to  the  teaching/learning  process. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 

SUBTASK:  COMPLETE  THE  CREATING  OF  SPECIFICATIONS  FOR  SEARCHING  THE 

INSTRUCTIONAL  VEHICLE  INVENTORY 


DESCRIPTION 

After  following  the  procedures  and  using  the  descriptor  listings  thus  far  in 
describing  the  innovation  as  it  might  relate  to  items  in  the  instructional 
vehicle  inventory,  it  may  be  useful  to  look  at  the  results  with  a view  of  add- 
ing commonsense  to  the  search  operation.  For  example,  if  the  devices  to  which 
the  innovation  applies  can  be  used  only  in  classroom  environments  (and  there 
are  no  formal  descriptors  which  distinguish  this  kind  of  training  environment), 
add  the  term  to  the  search  specification.  The  assessor  should  be  careful,  how- 
ever, not  to  include  a term  in  the  search  specification  that  will  be  improperly 
restrictive  and  will  risk  loss  of  a large  portion  of  relevant  devices  along 
with  the  elimination  of  a few  irrelevant  ones.  If  the  innovation  is  a promising 
one  for  whatever  vehicles  it  applies  to,  the  assessor  is  advised  to  follow  the 
safe  strategy  of  throwing  out  a wide  net  that  may  capture  a fairly  large  portion 
of  items  that,  on  his  manual  inspection  of  the  context  descriptions  of  the 
device,  he  recognizes  as  irrelevant  to  the  innovation  and  discards  as  candidates. 

PROCEDURE 

1.  Organize  and  set  down  all  descriptor  terms  that  are  relevant  to  selecting 
applicable  candidates  from  the  inventory  of  instructional  vehicles,  while 
rejecting  those  that  are  not  applicable.  The  descriptors  may  be  drawn 
from,  but  not  exclusively  from,  those  under  Vehicle  Type,  Vehicle  Property- 
Instructional  Function,  Vehicle  Property-Utilization  Function,  and  Class 

of  Training  Objective. 

2.  Check  the  descriptor  set  used  in  the  data  base  that  describes  each  entry 
in  the  listing  of  Instructional  Vehicles  for  further  clues  as  to  relevant 
descriptors  that  will  link  the  Innovation  to  appropriate  vehicles  and 
exclude  inappropriate  vehicles. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFICATION  OF  RELEVANT  INVENTORY  OF  TRAINING  VEHICLES 


PROCEDURE 

1.  Run  the  search  specification  summarized  in  Subtask  05.06  against  the  data 
base  of  the  complete  inventory  of  instructional  vehicles  used  by  the  Navy. 
The  search  operation  will  attempt  to  match  the  descriptors  in  the  search 
specification  with  the  descriptors  of  each  item  in  the  inventory  list. 

Those  that  match  will  be  presented  as  a listing  to  the  assessor.  Accompany- 
ing the  name  of  each  of  these  "hits"  will  be  an  abstract  or  summarized  text 
description  of  the  vehicle,  plus  a structured  but  detailed  description  of 
the  vehicle.  The  output  will  (or  should)  include  other  data  relevant  to 
the  benefits  assessment  to  be  made  in  the  next  step:  quantity,  range  of 
usage,  courses  and  course  objectives  used  for,  anticipated  future  demand 
rate,  and  various  categories  of  cost. 

This  output  listing  and  accompanying  contextual  descriptions  of  the  hits 
made  by  the  search  specification  is  only  the  first  stage  in  developing  a 
list  of  vehicles  relevant  to  the  innovation. 

2.  On  the  basis  of  a manual  inspection  of  the  context  descriptions  of  the 
candidate  vehicles  in  the  output  listing  reject  those  that  are  inappli- 
cable, and  accept  those  that  realistically  fit  the  full  characterization 
of  the  innovation. 

3.  Retain  the  listings  of  the  relevant  vehicles  and  the  context  data  associated 
with  each  of  them  for  Benefits  Assessment,  Subtask  05.08. 


\ 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFY  BENEFIT  PATTERN  FOR  INSTRUCTIONAL  VEHICLES 


DESCRIPTION 

The  assessment  of  benefits  in  this  path  of  analysis  must  be  limited  to  deter- 
mining the  cost  advantages  or  disadvantages  of  the  instructional  vehicle,  or 
Its  property,  to  which  the  innovation  is  relevant. 

There  are  two  considerations  in  assessing  the  benefit  pattern.  One  is  the  re- 
placement of  the  Instructional  vehicle  by  another,  which  may  be  the  innovated 
device  where  cost  is  the  sole  variable  of  Interest.  The  second  consideration 
Is  the  potential  benefit  that  may  be  obtained  by  using  the  innovation  as  im- 
pacting one  or  more  of  the  Utilization  Functions  (Subtask  05.05). 

PROCEDURE 

1.  If  the  benefit  Involves  the  replacement  of  one  vehicle  type  by  another, 
and  there  are  no  benefits  or  liabilities  associated  with  this  replacement, 
then  the  benefit  is  restricted  to  the  cost  differential.  No  further 
analysis  is  required  if  the  replacement  does  not  have  hidden  costs. 

2.  If  the  innovation  affects  certain  attributes  of  the  vehicle  for  which 
benefits  can  be  identified,  the  same  process  should  be  followed  that  is 
used  in  Subtask  05.18  by  using  the  "Format  for  describing  a supplementary 
benefit  or  liability." 

3.  If  the  benefit  pattern  of  displaceable  costs  does  not  appear  to  justify 
acceptance  of  the  Innovation  (which  would  permit  going  directly  to  Task  6) 
then  additional  benefits  should  be  explored  via  the  Task  Content  Training 
Route  (Subtask  05.09)  using  the  information  collected  in  Subtask  05.04. 


I I I -94 


TAEG  REPORT  NO.  12-3 

TASK  SUBTASK  PAGE 
05  09  01 


TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  DEVELOP  A BACKGROUND  FOR  TASK  CONTENT  TRAINING  ANALYSIS 


DESCRIPTION 

Assume  that  the  Innovation  points  to  a more  efficient  or  effective  set  of  opera- 
tions for  producing  a knowledge  or  skill  which  may  be  a training  objective  or  an 
on-the-job  performance  capability.  If  the  intended  outcome  can  be  related  to 
existing  training  objectives,  the  benefit  may  be  assessed  in  terms  of  a cost 
Improvement.  In  this  case,  the  evaluation  would  depend  on  the  courses  and  student- 
hours  affected  by  the  innovation  in  the  context  of  existing  objectives  on  student 
performance  levels. 

If,  however,  the  innovation  points  to  performance  benefits  that  go  beyond  existing 
training  objectives,  and  no  cost  improvements  can  be  demonstrated  from  the  innova- 
tion, the  analysis  must  be  applied  to  job  tasks  in  the  Navy's  operational  milieu. 

If  the  Innovation  increases  training  costs,  the  estimated  benefits  of  the  added 
performance  capabilities  must  be  assessed  as  worth  the  added  costs  or  not  worth 
the  added  costs. 

The  following  steps  for  determining,  by  means  of  task  descriptors,  what  training 
objectives  are  relevant,  also  apply  In  principle  to  determining  what  tasks  in 
the  operational  environment  are  relevant  — assuming  no  training  objective  exists 
that  is  relatable  to  the  operational  tasks  or  jobs  in  question. 

Assume  that  the  learning  outcomes  of  the  innovation  have  been  verbally  described. 
The  assessing  officer  must  probably  be  familiar  with  the  learning  processes  and 
outcomes  of  the  Innovation  In  operational  terms,  as  well  as  with  the  innovator's 
descriptions  of  conditions  and  outcomes.  This  substantive  knowledge  may  enable 
restrictions,  as  well  as  amplifications,  to  the  verbal  range-of-effect  perceived 
by  the  innovator.  Thus,  an  innovation  that  the  innovator  perceives  as  appli- 
cable only  to  the  medical  student's  learning  of  the  names,  positions,  and  actions 
of  muscle  and  bones  in  the  body,  may  be  generalized  to  the  integration  of  in- 
struction on  nomenclature  and  identifications  of  anj£  sets  of  work  objects,  and 
also  for  Instruction  in  the  verbal  stages  of  learning  sequential  procedures  of 
any  kind.  The  following  sets  of  categorizations  should  foster  the  making  of 
"tTSTs  kind  of  generalizing  insight,  as  well  as  to  place  proper  restrictions  and 
qualifications  on  the  range  of  applicability. 

PROCEDURE 

1.  Become  familiar  with  the  conditions  for  selecting  the  task  content  analysis 
route  for  determining  the  range-of-effect  for  the  innovation. 

2.  Study  the  description  of  the  innovation  in  Task  1;  the  statement  of  task 
objectives,  training  objectives,  and  especially  the  context  of  the  empirical 
findings  as  reported  in  Task  1. 
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3.  Examine  the  terms  in  the  various  subtasks  from  05.12  through  05.21.  Some 
one  or  combination  of  these  terms  (and  their  reference  definitions  in  the 
text  describing  the  subtasks)  may  suggest  a more  direct  way  of  describing 
the  innovation  for  searching  through  the  inventory  of  Navy  training  courses 
and  the  inventory  of  Navy  job  tasks,  than  the  formal  route  structure  might 
require.  Notice  that  each  choice  of  a descriptor  along  the  route  both  opens 
a corridor  of  relevant  applicability  of  the  innovation,  and  at  the  same 
time,  closes  off  other  potential  corridors  as  irrelevant  to  the  innovation. 

4.  Do  not  be  limited  to  the  description  of  the  tasks  and  courses  by  any  ter- 
minology appearing  in  the  subtask  analyses.  If  you  can  confidently  go 
directly  to  the  names  of  all  the  courses  clearly  relevant  to  the  innovation, 
do  so.  Or,  if  there  are  other  terms  which  will  more  quickly,  and  compre- 
hensively, identify  the  jobs  and  tasks,  or  courses,  use  them. 

5.  If  the  innovation  is,  without  doubt,  limited  to  an  already  Identified  job 
and  task  within  the  job,  or  an  identifiable  course  and  course  objective, 

it  Is  unnecessary  to  go  through  an  elaborate  search  operation.  Go  directly 
to  Subtask  05.18,  Assess  the  Benefit  Pattern.  It  is  possible,  however,  that 
the  Innovation,  either  as  instruction  or  as  a result  in  a skill  level,  may 
be  applicable  to  a class  of  courses  and  jobs  with  which  the  assessor  is  not 
familiar.  The  data  base  search  on  selected  descriptors  would  then  be  a 
safe  hedge  against  this  unfamiliarity. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFY  THE  LEARNING  OBJECTIVE 


DESCRIPTION 

The  primary  source  of  information  for  specifying  the  learning  objective(s) , 
preliminary  to  more  formal  analysis,  is  the  description  of  the  innovation 
made  in  Task  1.  That  description  may  be  supplemented  and  qualified  inform- 
ally by  training  experts  who  have  operational  knowledge  about  the  innovation 
and  any  studies  performed  with  it,  or  more  detailed  knowledge  about  the  jobs 
and  tasks  targeted  by  the  innovation  than  was  known  to  the  innovator. 

PROCEDURE 

Modify  or  extend  the  description  of  the  innovation  as  appropriate  about  the 
training  product  or  process  to  which  it  applies,  and  the  task  and  task  context 
to  which  it  applies. 

COMMENT 

The  procedures  in  range-of-effect  will  result  in  further  analysis  of  the  innova- 
tion, and  some  of  those  results  may  be  redundant  to  what  is  done  in  this  sub- 
task. However,  since  the  description  of  the  innovation  made  here  will  be  a 
major  starting  point  and  reference  for  further  analysis,  the  completeness  of 
the  description  made  here,  including  restrictions  and  qualification,  should 
facilitate  later  steps.  * 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  MAKE  PRELIMINARY  EVALUATION  OF  BENEFIT  IMPORTANCE 


DESCRIPTION 

This  sub task  has  two  major  parts: 

a.  Is  the  training  objective  to  which  the  innovation  is  limited  of  minor, 
moderate,  or  essential  importance?  If  the  innovation  is  directly  tied 
to  a class  of  task  such  as  the  rigging  of  models  of  sailing  ships,  it 
may  be  possible  to  discard  the  proposal  quickly.  Note  whereas  an  out- 
of-hand  rejection  of  an  innovation  at  this  point  might  save  further 
assessment  work,  it  could  lead  to  overlooking  a significant  instructional 
principle.  In  general,  instruction  in  a nonskill  directed  knowledge 
should  have  less  importance  than  skill  training. 

b.  Is  the  magnitude  of  the  found  or  potential  improvement  in  training  benefit, 
or  achieving  the  training  objective,  small,  moderate,  or  very  substantial? 
When  humans  must  convert  a principle  or  reference  model  into  actualities 

in  their  own  contexts,  a substantial  degradation  of  the  potential  benefit 
of  the  principle  tends  to  occur.  A benefit  of,  say  20%  shown  in  a labora- 
tory, may  shrink  to  just  a few  percent,  or  zero,  when  applied  by  average 
practitioners.  Exceptions  may  occur. 

PROCEDURE 

1.  Examine  carefully  the  description  of  the  innovation,  and  the  extended 
description  of  the  kinds  of  training  objective  towards  which  the  innova- 
tion appears  restricted,  as  described  in  Subtask  05.10.  Take  into  account 
the  learning  requirements  of  the  student  in  his  pattern  of  learning  and 
fulfillment  of  training  requirements , but  consider  even  more  seriously  the 
importance  of  the  instructional  objective  (as  characterized  thus  far). 
Select  the  answer  which  best  fits  your  conclusion: 

The  training  objective  to  which  the  innovation  is  targeted  seems, 

in  terms  of  the  training  mission  and  operational  mission  of  the 

Navy,  of: 

a.  Identifiable  but  minor  importance 

b.  Moderate  importance 

c.  Very  substantial  importance 

d.  Crucial  importance. 
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2.  Examine  the  empirical  data  or  other  basis  for  estimating  the  magnitude  of 
the  Incremental  benefit  in  training  value  of  the  innovation  as  compared 
with  the  existing  or  reference  condition.  Take  into  account  not  only 
data  averages  but  their  distributions  (standard  deviations).  Select  the 
answer  which  best  fits  your  conclusion: 

The  magnitude  of  the  benefit,  taking  into  account  all  factors  that 

could  have  practical  bearing  on  the  results,  is: 

a.  Of  no  practical  importance 

b.  Of  identifiable  to  moderate  importance 

c.  Of  very  substantial  Importance 

d.  Of  almost  crucial  Importance. 

3.  A low  or  negative  score  on  both  of  these  questions  could  lead  to  an  imme- 
diate termination  of  the  assessment  process  with  a rejection  of  the  inno- 
vation. However,  even  though  the  assessor  had  well  in  mind  the  major 
variables  in  training  and  in  on-the-job  performance  requirements,  and  he 
ran  this  internalized  model  against  his  understanding  of  the  innovation, 
the  risk  of  losing  a good  bet  might  be  small,  but  not  nonexistent. 

If  the  Innovation  is  rejected  at  this  point,  the  work  done  thus  far  should 

be  documented  and  entered  in  the  data  base  of  rejected  innovations. 

4.  If  the  innovation  passes  through  this  test,  proceed  into  the  determination 

of  range-of-effect  and  formal  benefits  analysis.  Proceed  to  Subtask  05.12, 
Determine  If  the  Benefit  Applies  to  Skill  or  Knowledge  Training. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  DETERMINE  IF  THE  BENEFIT  APPLIES  TO  SKILL  OR  KNOWLEDGE  TRAINING 


DESCRIPTION 

Determining  range-of-effect  for  knowledge  training  is  a different  route  than 
range-of-effect  for  skill  training.  The  descriptors  along  each  route  should 
be  the  primary  basis  for  the  selection  of  the  route.  These  descriptors  are 
intended  to  provide  the  basis  for  the  distinction.  Skill  training  is  actual 
practice  on  an  operational  work  task,  in  actual  or  simulated  conditions.  Note 
that  different  stages  of  skill  learning  may  call  for  different  learning  condi- 
tions. A skill  is  a doing  or  ability  to  do;  it  has  specific  references  in 
equipment  and  other  work  objects,  environments,  tools,  reference  information, 
and  criteria  of  goodness.  Skills  may  be  symbolic  and  cognitive  such  as 
decision-making  as  well  as  manipulative,  procedural  as  well  as  tracking. 
Training  that  is  not  skill-oriented  is,  by  definition,  "knowledge"  oriented. 

It  should  be  repeated  that  making  the  classifications  that  follow  is  not  an 
end  in  itself,  but  a means  to  find  a set  of  descriptors  that  fit  the  range  of 
training  objectives  and  training  and  learning  processes  appropriate  to  the 
innovation's  potential  utility.  The  necessary  and  sufficient  training  supports 
for  the  learning  of  a tracking  skill,  for  example,  differ  from  those  generally 
necessary  and  sufficient  to  teach  and  learn  a procedural  skill,  which  in  turn 
differ  from  decision-making  and  constructing  skills.  Each  category  implies  a 
training  resource  that  differs  in  kind  or  magnitude  for  a training  resource 
both  efficient  and  effective  for  another  category  of  skill.  This  statement 
applies  mainly  to  the  difference  between  skill  and  knowledge  pathways,  and  to 
terms  listed  under  Task  Structure  Element,  Task  Formats,  and  Stage  of  Learning. 

Distinctions  are  made  here  between  "Reference  Knowledges"  and  "Enabling  Know- 
ledges." The  latter  are  skill-oriented  but  may  be  acquired  through  cognitive 
and  symbolic  processes,  rather  than  by  actual  practice  on  the  task  itself  in 
some  real  or  simulated  task  setting. 

The  distinctions  summarized  here  are  clarified  in  the  body  of  the  respective 
topics. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  TASK  STRUCTURE  ELEMENT  OR  TASK  "FUNCTION" 


DESCRIPTION 

This  Is  another  step  in  selecting  descriptors  that  will  pin  down  the  applica- 
bility of  the  innovation,  while  at  the  same  time  probing  for  the  extent  of 
its  range. 

A skill,  generally,  is  made  up  of  a number  of  components.  Although  there  are 
many  ways  of  classifying  or  subdividing  components,  the  set  of  descriptors 
used  here  is  related  to  instructional  and  learning  properties.  Each  task 
element  denotes  a class  of  capability  that  has  somewhat  different  demands  on 
what  is  necessary  in  an  instructional  capability,  than  is  required  for  another 
task  element.  Thus,  clearly,  learning  to  scan  a piece  of  equipment  and  detect 
burned  insulation,  broken  wires,  or  leaking  receptacles,  demands  a lower  mini- 
mum effective  instructional  capability  than  would  be  essential  to  learning  to 
i denti fy  the  damaged  wire  or  leaking  receptacle,  or  interpret  the  cause  of  the 
damage  or  its  effect  on  the  performance  of  the  device. 

In  this  sub task,  we  are  trying  to  narrow  the  relevant  applicability  of  the 
innovation.  The  result  of  this  step  will  have  sharpened  the  description  of 
the  conceptual  applicability  of  the  innovation,  in  readiness  to  apply  it  to  a 
search  of  the  job  task  inventory  for  reasonable  matches  to  the  specification 
developed  thus  far. 

The  name  of  the  relevant  task  structure  element,  in  itself,  can  sometimes  serve 
as  the  principal  search  description  applied  to  an  inventory  of  training  courses 
and  training  objectives,  or  to  an  inventory  of  task  descriptions.  Thus,  a train- 
ing technique  for  the  "Identification  of  figural  elements  in  complex  perceptual 
fields"  may  apply  to  a skill  in  "identifications"  from  reconnaissance  photos  to 
visual  checkout  of  an  electronic  chip  layout. 

The  following  is  a brief  list  of  what  may  be  called  task  structure  elements  or 
functions.  Although  the  set  is  small,  it  covers  a wide  range  of  real-life  tasks, 
and  especially  if  the  items  are  taken  in  combination.  Thus,  scan-and-detect 
may  be  combined  with  identify,  identify  with  interpret,  interpret  with  decision- 
making, and  so  on,  in  order  to  match  with  the  training  or  job  objective  in  the 
innovation. 

The  following  points  should  be  clearly  understood: 

• Every  work  activity  will  be  represented  to  some  degree  by  all  of  these  task 
functions.  But  in  most  tasks  as  performed  in  a given  situational  context, 
only  some  one  or  several  will  predominate  and  be  the  critical  training 
factor. 
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• Each  of  these  functions  has  a learning  and  training  requirement  that 
differs  somewhat  from  each  of  the  other  functions. 

• It  is  possible  for  learning  to  be  achieved  in  each  of  these  task  functions 
independent  of  learning  on  other  task  functions,  and  at  suitable  stages  of 
learning,  to  be  effectively  integrated  with  what  has  been  learned  indepen- 
dently on  other  task  functions  or  elements.  This  ability  to  combine  part- 
learning efficiently  into  whole  patterns  of  ability  does  not  apply  to  any 
randomly  conceived  subsetting  of  behavior  and  performance  ‘‘elements". 

In  this  subtask,  the  set  of  task  structure  elements  of  major  utility  will 
merely  be  listed,  and  brief  identifying  descriptions  will  be  given.  The  two 
exceptions  are  Procedure  Following  and  Tracking.  In  Appendix  A,  most  of 
these  terms  are  given  a more  detailed  and  analytic  treatment,  and  implications 
for  training  will  be  spelled  out.  The  assessor  is  advised  to  become  familiar 
with  task  structure  in  the  present  context  because  of  the  power  it  can  confer 
in  the  design  of  highly  efficient  and  cost-sensitive  systems  of  overall  skill 
training. 

The  list  of  task  structure  elements  consists  of: 

Goal  Projection 
Scan-Detect 
Identify 
Interpret 

Procedure  Following 
Decide 

Construct/Plan 

Tracking 

Motor  Performance 
Interpersonal  Interaction 
Recall  Task-Cycle  Information 
Recall  Enabling  Information. 

These  elements  are  briefly  defined  on  the  following  pages  under  Procedure,  Step  1. 
Most  of  them  are  treated  more  extensively  in  Appendix  a. 

PROCEDURE 

1.  After  careful  study  of  the  description  of  the  target  tasks  in  the  innova- 
tion in  the  light  of  the  definitions  of  the  task  structure  elements  given 
below  and  in  Appendix  a*  select  the  one  or  more  task  elements  which  apply 
to  the  innovation. 


• Goal  Projections:  Projecting  a mental  picture  of  what  the  result  of  a good 

performance  of  the  job  or  task  under  given  circumstances  should  be.  This 
picture,  if  complete,  would  include  a representation  of  all  relevant 
criteria  of  effective  completion.  It  is  an  image  of  what  the  outcome  of  a 
given  performance  should  achieve.  It  should  be  realistic  and  practical  in 
terms  of  a competent  performer  acting  in  a context  of  concrete  circumstances 
The  goal  projection  serves  to  initiate  task  behavior,  direct  and  sustain  it 
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to  a work  cycle  conclusion.  A test  of  the  validity  of  the  operator's  goal 
projection,  other  than  in  his  performing  the  task  itself,  is  his  ability  to 
discriminate  an  adequate  from  inadequate  representations  of  proper  goal 
states,  given  a concrete  set  of  conditions  under  which  the  task  is  to  be 
performed.  If  the  operator  has  a clear  and  complete  goal  image,  it  is 
possible  for  him  to  compare  the  results  of  his  actual  performance  with  the 
results  of  criterion  performance,  note  the  deviations  and  thus  give  him- 
self "learning  feedback"  which  he  can  use  to  improve  his  performance  when 
he  performs  the  task  again.  The  goal  image  thus  serves  the  dual  purposes 
of  directing  performance  and  of  directing  effort  in  skill  improvement. 

• Scan-Detect:  Scanning  a work  environment,  using  appropriate  senses,  and 

detecting  the  presence  of  cues  or  conditions  requiring  some  .form  of  action 
to  be  taken.  The  action  may  consist  of  further  examination  of  the  cue, 
leading  to  an  identification  or  to  assisting  an  interpretation;  or  an 
inmediate  overt  action  (such  as  avoiding  collision  with  an  obstacle)  may 

be  the  result.  Learning  to  scan-detect  In  a given  task  context  is  learning 
what  to  pay  attention  to  and  notice;  in  complex  tasks  It  will  consist  of 
suitable  divisions  of  attention  among  sources  of  cues.  In  some  tasks, 
scanning  technique  may  require  sophisticated  search  strategies,  or  emphasize 
sustained  vigilance.  There  are  tasks  in  which  the  detecting  component  is 
made  difficult  because  relevant  cues  are  obscured  by  noise,  irrelevance  or 
distractions. 

• Identify:  To  differentiate  an  object  or  process  or  condition  from  others 
in  a set,  and  generally  put  a name  or  label  to  it.  Or,  conversely,  given 
the  name,  to  point  to  the  applicable  object  or  condition.  T©  identify  may 
be  not  only  to  differentiate,  as  one  hamper  from  another,  but  to  generalize 
within  a class  or  set  of  entities  such  as  "hammers".  Being  able  to  apply 
verbalized  job  instructions  demands  the  incumbent's  ability  to  link  the 
appearance  of  the  object  (or  the  making  of  the  operation)  to  the  matching 
word  or  expression.  In  some  tasks,  such  as  target  identification  in  radar 
displays  or  in  sonar  presentations,  the  skill  itself  is  in  the  perceptual 
process  of  detecting  and  organizing  cues  that,  as  a pattern,  will  yield  a 
recognition  of  an  Identity. 

• Interpret:  Responding  to  one  or  a group  of  cues  as  a pattern  or  context 

that  has  meaning  where  the  meaning  Is  a cognitive  Inference.  The  letter 
symbols  "cat"  will  tend  to  have  a different  significance  or  meaning, 
depending  on  whether  they  occur  in  a discussion  about  domestic  animals, 
earth-moving  equipment,  or  malicious  women.  Interpretation  requires  adding 
information  to  that  which  is  physically  presented  as  individual  cues.  It 
may  also  require  responding  to  a collection  of  separate  cues  as  a pattern 
or  gestalt.  Interpretations  may  be  Incorrect:  a power  cord  out  of  the 

power  source  socket  may  "mean"  that  the  appliance  is  not  in  operation  -- 
but  only  if  the  appliance  lacks  some  other  power  source.  The  output 
actions  in  some  tasks  may  depend  heavily  on  the  Incumbent's  ability  to 
respond  to  patterns  of  cues  in  terms  of  what  they  reference  or  mean 
rather  than  to  their  individual  presence  or  absence.  In  perceptual  tasks, 
an  example  is  reconnaissance  photo  interpretation.  In  conceptual,  cognitive 
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tasks,  an  example  is  tactical  or  strategic  planning  on  the  partial  basis  of 
signs  by  the  energy  that  are  interpreted  as  his  "Intentions". 

t Procedure  Following:  Carrying  out  a sequence  of  discrete  actions  or  steps 

in  a more  or  less  invariant  order  except  at  conditional  branch  points.  The 
sequence  of  steps  In  some  classes  of  procedure  is  more  fixed  by  prescription 
than  In  others.  The  procedure  is  ordinarily  initiated  by  an  explicit  demand 
signal  in  a work  context  and  terminated  by  a specified  final  action  step 
or  identification  of  a goal  state.  Each  procedural  step  is  characterized 
by:  (a)  a stimulus  state  or  specified  cue  appearing  on  one  or  more  Indi- 

cators or  physical  objects;  (b)  a specified  task  action;  or  (c)  a result 
of  the  task  action  appearing  as  a specified  cue  of  response  adequacy  or 
inadequacy;  this  feedback  cue  may  be  the  stimulus  for  taking  (or  selecting) 
the  next  step  In  the  procedure. 

The  set  of  steps  to  be  taken  in  a radar  checkout  or  alignment  operation 
constitute  a procedure.  This  is  an  example  of  a highly  prescriptive  pro- 
cedure; each  step  is  well-marked,  the  sequence  is  fixed,  the  cue  for  each 
step  is  explicit  and  unambiguous.  (The  exceptions  may  require  an 
"Interpret"  task  element.) 

On  the  other  hand,  the  set  of  steps  to  be  taken  by  a surface  vessel  in 
locating  the  position  of  a suspected  enemy  submarine  may  be  a highly 
strategic  procedure:  although  a principle  specifies  what  general  steps 

are  to  be  taken  and  in  what  order,  each  step  is  largely  conditional  on 
environmental  context  and  on  the  results  of  previous  steps.  More  informa- 
tion processing  is  required  to  perform  strategic  procedures  than  to  per- 
form prescriptive  procedures,  therefore,  the  latter  require  less  training 
time  and  facility  where  other  factors  are  held  equal.  Training  for  strategic 
procedures  overlaps  considerations  for  decision-making  training. 

Special  recognition  needs  to  be  made  about  activities  loosely  called 
"Communicating  Verbal  Information."  In  some  work  settings,  the  structure 
of  the  communications  is  highly  standardized.  One  example  is  that  of  the 
control  tower  personnel  interacting  with  pilots.  Another  is  that  of  the 
mechanic  filling  out  a maintenance  form  detailing  the  work  he  performed. 

These  are  examples  of  procedural  formats,  and  may  be  taught  and  learned  as 
procedures.  Remember  that  learning  a procedure  should  also  extend  to  learn- 
ing to  cope  with  contingencies  associated  with  the  procedure.  A second 
class  of  "communicating  verbal  Information"  requires  the  more  or  less  free 
form  construction  of  a narrative  or  scenario:  the  technician's  explanation 

of  "why  the  flying  of  a mission  with  an  airplane  having  a given  defect  would 
be  dangerous"  would  be  an  example.  The  task  element  rubric  "Construct"  would 
apply  to  the  latter  as  an  indicator  of  training  and  of  performance  character- 
istics. But  where  the  rational  and  objective  factors  must  be  supplemented  by 
skills  In  perceiving  and  formulating  responses  to  emotional  and  attltudinal 
factors  in  interpersonal  communication,  a new  dimension  in  task  requirement, 
aptitude,  training  and  performance  Is  Introduced.  This  dimension  Is 
acknowledged  in  the  separate  task  element  called  "Interpersonal  Interaction." 
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Procedures  may  combine  prescriptive  and  strategic  components.  The  improvised 
welding  of  hull  plates  at  sea  would  provide  many  examples.  Using  heavily 
damaged  engines  to  bring  a ship  Into  port  would  inevitably  combine  the  pres- 
criptive with  the  strategic. 

Learning  to  execute  a procedure  is  generally  preceded  by  cognitive  phases  in 
which  the  component  stimulus  indicators  and  Indications  are  identified  and 
located  and  named,  as  well  as  the  objects  (controls)  on  which  the  student's 
response  is  to  be  effected.  This  enables  the  student  to  use  verbal  or 
diagranmatlc  information  to  formulate  a conceptual  model  (or  picture  in  the 
head)  of  the  sequence  of  cues  and  actions  making  up  the  procedure.  This  con- 
ceptual picture,  If  It  Is  sufficiently  clear  and  the  student  is  able  to  trans- 
late it  Into  perceptions  and  actions,  can  guide  student  performance.  This 
cognitive  Image  may  be  supplemented  during  practice  by  job  manuals  that  de- 
scribe the  procedure. 


Retention  and  recall  of  a procedure  is  poorest  when  the  procedure  has  been 
learned  to  an  Intermediate  degree.  This  Is  the  level  at  which  the  student 
may  have  learned  many  of  the  behavior  components,  but  not  their  relation- 
ships and  sequences,  either  cognitively  or  in  perceptual -motor  behaviors. 

This  level  may  be  evidenced  by  an  appearance  of  "bare  mastery"  of  the  pro- 
cedure such  as  in  an  early,  stumbling  carry- through  of  the  procedural 
scenario.  (Unfortunately,  much  formal  training  for  many  students  terminates 
at  about  this  point  In  learning  level,  and  further  practice  In  the  task 
resumes  on  the  job  Itself  where  procedural  errors  may  be  difficult  to  Identify 
and  eliminate.)  Retention,  as  measured  by  recall  and  savings  In  relearning, 
tends  to  be  high  when  the  student,  by  sufficient  repetitive  practice,  has 
integrated  the  procedural  components  into  a smooth  sequencing  of  perceptions 
and  actions.  At  this  stage,  he  can  be  Interrupted  and  then  resume  readily 
where  he  left  off;  he  can  anticipate  what  step  he  will  be  taking  next;  he 
can  retain  Information  specific  to  a given  task  cycle.  In  this  stage,  he 
has  learned  not  merely  a chain  of  otherwise  Isolated  behavior  elements,  but 
a context  of  associated  behaviors  that,  as  a combination,  resists  forgetting 
and  interference  Influences. 


Decide:  Selecting  a course  of  action  by  systematic  processing  of  Informa- 

tion about  outcome  requirements  and  the  situation  on  the  one  hand,  and 
response  alternatives  and  strategies  on  the  other.  Choices  are  usually 
made  In  some  degree  of  uncertainty  about  the  actual  situation,  the  need, 
and  how  a given  choice  will  work  out.  For  example,  under  marginal  weather 
conditions  and  a given  importance  of  mission  and  a given  aircraft  with  per- 
formance limitations,  to  choose  whether  to  fly  the  mission  or  not,  assuming 
an  option  for  choice  exists.  In  general,  a decision  differs  from  a straight- 
forward procedure  In  that  all  of  the  Information  necessary  and  sufficient 
for  choosing  an  action  Is  not  available,  or  is  ambiguous,  so  that  probability 
and  risk  (In  some  degree)  are  Involved  in  selecting  an  action. 

Note  that  where  specific  rules  cover  the  selection  of  the  "correct"  response 
from  potential  alternatives,  the  operator  learns  a procedure.  The  procedure 
may  be  complex,  but  is  nonetheless  a procedure  and  may  be  learned  by  the 
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mechanics  of  procedure  learning.  There  may  be  occasional  gray  areas 
between  a task  that  is  performed  as  a procedure  or  that  may  be  per- 
formed as  a decision-making  (or  "judgmental")  operation.  Even  in  the 
latter  cases,  however,  the  separation  of  the  procedural  format  from 
the  decision-making  format  for  managing  uncertainty  may  be  useful  for 
instructional  and  performance  purposes. 

The  greater  the  number  of  variables  that  should  be  considered  in 
reaching  a choice,  the  more  difficult  it  is  to  learn  to  make  the  deci- 
sion proficiently.  The  multiple  variables  may  characterize  the 
definition  of  the  problem  situation,  or  multiple  options  may  characterize 
the  choice  of  the  response,  or  both  the  situation  and  the  resources  for 
response  selection  may  be'  complex.  Because  the  selection  of  the  response 
is  made  on  a probabilistic  basis,  it  is  generally  impossible  to  be 
absolutely  rigorous  in  evaluating  the  student's  choice,  except  where  an 
Ineffective  choice  occurred  because  of  neglecting  a key  variable  in  the 
situation  or  a key  variable  In  the  resource  committed  to  a response. 

The  decision  maker  learns  to  cope  with  the  variables  In  the  problem 
class;  he  must  also  learn  to  cope  with  the  kinds  and  levels  of  un- 
certainty inherent  to  the  class  of  decision  he  makes. 

• ' Construct/Plan:  Creating  a response  pattern  that  tends  to  be  unique  to 

a unique  pattern  of  conditions  and  objectives  and  resources.  But  the 
variables  in  the  kind  of  construction  that  Is  made  (such  as  a table  of 
organization)  or  in  a plan  (such  as  planning  a landing  operation),  tend 
to  be  constant  from  one  case  to  another.  (The  variables  in  planning  a 
landing  operation  tend  to  be  about  the  same  from  one  landing  to  another.) 

• Tracking:  Responding  to  a more  or  less  continuously  changing  stimulus 

by  more  or  less  continuous  changes  in  response  action,  having  the  critical 
requirements  of  smoothness,  coordination  and  timing.  A given  tracking 
skill  is  the  manifestation  of  the  operator's  anticipation  of  the  dynamic 
relationship  between  target  data  (the  movement  of  the  "cursor")  and  given 
actions  on  the  controls  he  manipulates.  In  pursuit  tracking,  such  as 
flying  an  aircraft  visually  towards  a moving  or  stationary  target,  the 
operator  can  differentiate  the  dynamics  of  the  target  from  the  dynamics 
of  the  device  he  operates  with  controls  (his  aircraft).  In  compensatory 
tracking,  such  as  in  blind  flying  to  the  aircraft's  attitude  indicator, 
the  dynamics  of  the  target  (such  as  air  turbulence)  are  not  readily 
differentiable  In  the  displayed  Information  from  the  dynamics  of  the 
control  instrument  (the  aircraft's  responding  to  control  manipulations). 
This  confounding  of  dynamic  characteristics  of  the  external  target  with 
the  dynamic  characteristics  of  the  controlled  instrument  or  vehicle 
makes  learning  to  anticipate  their  interaction  very  difficult;  compensa- 
tory tracking  is  notoriously  more  difficult  to  learn  than  pursuit  tracking. 


Piloting  a vehicle  differs  from  aiming  a gun  at  a target.  A vehicle  must 
be  moved  through  apertures  that  the  pilot  must  project  ahead,  to  the  side, 
and  even  to  the  rear  if  he  is  concerned  with  safety.  The  aperture  must 
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be  large  enough  to  include  tolerances  for  the  pilot's  skill,  for 
probabilities  of  invasion  of  the  projected  aperture  by  obstacles, 
either  from  the  sides  or  on  the  surface  of  the  roadway.  Maneuvering 
a vessel  has  equivalent  variables  and  requirements.  The  skill  of  the 
operator  depends  as  much  on  his  ability  to  project  the  behavior  of 
aperatures  in  his  operating  environment,  as  on  his  ability  to  project 
the  behavior  of  the  vehicle  in  response  to  the  control  dynamics  he 
applies  to  accelerator,  brake,  and  steering  wheel. 

The  technology  of  instruction  and  learning  of  tracking  tasks  is  an  ex- 
tensive literature  not  to  be  reviewed  here.  There  is  little  that  can 
be  learned  from  verbal  materials  that  transfer  to  tracking  tasks. 

Tracking  must  be  learned  by  doing.  On  the  other  hand,  there  are  many 
examples  of  simulators  that  seek  levels  of  physical  "fidelity"  in  dis- 
play-control dynamics  that  are  unwarranted  logically  and  psychologically, 
but  achieved  at  very  high  cost. 

In  real-life  tasks,  the  operator  must  generally  attend  to  multiple 
streams  of  information  at  the  same  time.  To  do  this  effectively  requires 
large  amounts  of  practice.  The  performance  of  procedures  is  often  mixed 
with  tracking  activities.  To  the  extent  that  the  procedure  does  not 
depend  on  information  contexts  outside  the  procedure  itself,  these  pro- 
cedures may  be  efficiently  learned  to  high  degrees  of  mastery  on  pro- 
cedural trainers  before  the  student  practices  them  in  full  work  context  — 
that  is,  on  a high-fidelity  simulator  where  the  operator  must  make  suit- 
able divisions  of  his  attention. 

It  should  be  clear  that  an  understanding  of  the  learning  process  enables 
the  formulation  of  a training  schedule  or  scenario  which  maximizes  the 
learning  and  transfer  of  training  benefits  from  a family  of  instructional 
devices,  each  of  which  may  cost  an  order  of  magnitude  less  than  the  next 
higher  in  functional  capability.  This  issue  relates  both  to  Stage  of 
Learning  and  to  part-task  training  considerations. 

• Motor  Performance:  Executing  a pattern  or  sequence  of  motor  activity 

where  the  emphasis  on  the  totality  and  its  parts  is  on  dexterity,  control, 
strength  and  precision  of  output.  The  major  concern  is  on  differentia- 
tion and  organization  of  muscle  groups  into  performance  patterns. 

Examples:  typing  and  other  keying  operations,  hand-to-hand  combat,  row- 

ing a boat,  sports  in  general.  In  a widely  different  context,  control 

of  the  voice  in  speaking,  singing  or  in  learning  to  pronounce  in  a 
foreign  language. 

• Interpersonal  Interaction:  Apprehending  and  presenting  information  to 

other  individuals  in  a manner  that  facilitates  getting  a collective  job 
done  by  the  participants.  This  task  function  connotes  more  than  pro- 
cedural factors  in  the  sense  of  job  manual  instruction.  It  includes 
personality  elements,  attitudes,  interpersonal  predispositions  and 
style  --  and  other  concepts  that  are  difficult  to  reduce  to  operational 
definition  except  indirectly,  such  as  by  the  observation  of  performance 
consequences.  It  is  outside  the  scope  of  the  present  project  to  develop 
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analytic  structures  and  rationales  that  apply  to  the  learning  and  ex- 
pression of  interpersonal  skills.  It  is  relevant,  however,  to 
acknowledge  them  as  a special  and  important  class  of  activities  in  any 
work  environment,  and  to  recognize  that  in  some  jobs  they  may  be  as 
important  than  technical  competences:  intimate  leader- follower  re- 

lationships and  team  coordination  situations  are  examples. 

e Recall  Task-Cycle  Information:  Remembering  and  applying  data  peculiar 

to  a work  cycle  and  relevant  to  the  activities  performed  in  that  work 
cycle.  Examples:  remembering  what  one  has  already  done  and  not  done 

In  carrying  out  a procedure;  a pre-mission  briefing;  a special  operating 
condition  such  as  a device  having  an  intermittent  fault;  the  temporary 
exposure  of  a live,  high  tension  wire.  The  process  of  being  aware  of 
what  has  been  completed  and  not  completed  while  performing  a procedure 
depends  on  this  memory  function.  It  is  necessary  to  hold  in  mind  the 
information  acquired  over  time  that  enters  into  the  making  of  a decision. 
In  psychological  jargon,  this  task  function  may  be  called  "working 
memory"  or  "short  term  memory."  In  Appen.dlx  A,  the  name  used  is 
"short  term  memory".  Its  operation  is  obviously  essential  to  the  per- 
formance of  any  tasks  above  the  most  primitive  conditioned  response 
levels.  Clearly  there  are  training  and  learning  techniques  that  can 
enhance  or,  on  the  other  hand,  depress  the  effectiveness  of  this  function. 
Reliable  recall  of  task  cycle  information  depends  on  the  learning  and 
exercise  of  task  formats  as  described  in  Task  05,  Subtask  23. 

• Recall  Enabling  Information:  Remembering  and  applying  facts,  rules  and 

conceptual  principles  applicable  to  task  performance.  The  classes  of 
information  that  fit  here  have  been  previously  identified  as  Enabling 
Knowledges.  Enabling  knowledge  ranges  from  the  stall  speed  of  the  air- 
plane one  is  flying  to  the  Centigrade  equivalent  of  70  degrees  Fahren- 
heit or  the  shank  diameter  of  a number  six  wood  screw.  Correctly 
recalling  such  Information  in  response  to  a classroom  test  is  a different 
condition  for  performance  from  that  of  recalling  the  same  Information 
under  the  stress  of  operational  conditions.  In  the  Appendix  A treatment 
of  this  topic,  the  concept  of  this  long-term  memory  is  called  "Store" 

In  order  to  distinguish  It  from  "Short  Term  Memory".  It  should  be 
recognized  that  the  performance  of  Procedures  implicitly  Implies  the 
recall  of  Information  that  supports  procedural  performance.  But  be- 
cause the  learning,  remembering  and  applying  of  support  Information 
can  have  aptitude  and  training  requirements  that  differ  from  the  learn- 
ing of  procedural  operations,  as  such,  the  recall  of  enabling  informa- 
tion justifies  its  own  Identity  and  characterization  as  a task  function. 
Furthermore,  if  two  tasks  differ  in  procedural  operations,  but  demand 
similar  classes  of  enabling  and  support  information,  there  should  be 
substantial  transfer  of  training  from  one  to  the  other  that  Is  evident 
after  the  early  stages  of  procedural  learning  in  the  second  task. 

Retain  the  names  of  the  relevant  task  structure  elements  and  any  information 

context  about  them  that  can  be  useful  for  the  next  stages  of  determining 

range-of -effect.  In  many  cases,  this  subtask,  of  selecting  task  structure 
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element(s)  will  merge  Into  the  Subtask  05.14,  Identifying  relevant  jobs 
and  tasks  In  the  Navy  Inventory  of  jobs  and  tasks. 

3.  Proceed  to  Subtask  05.23  to  determine  If  the  Innovation  Is  specifically 
applicable  and  restricted  training  In  task  formats. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFY  RELEVANT  JOBS  AND  TASKS 


DESCRIPTION 

The  key  objective  in  determining  the  range  of  the  innovation's  effect  will  be 
achieved  in  this  subtask,  assuming  that  training  courses  will  be  readily  iden- 
tified when  the  specific  job  tasks  have  been  identified.  In  all  cases,  the 
assessing  officer  is  advised  to  think  beyond  the  surface  matching  of  words  used 
to  describe  tasks  and  attempt,  rather,  to  match  on  the  processes  and  things  that 
lie  behind  the  words  used  to  describe  them.  "Disassembling"  a pocket  watch  and 
"disassembling"  a steam  turbine  may  have  some  points  of  overlap  in  terms  of  their 
respective  skills,  and  the  Instructional  operations  for  acquiring  them,  but 
their  differences  are  likely  to  be  greater  than  their  similarities.  On  the 
other  hand,  similarities  in  the  names  of  objects  or  of  tasks  are  good  hypotheses 
for  looking  further  into  the  relevant  similarities. 

Finally,  however,  the  decision  of  relevance  or  non-relevance  of  the  innovation 
to  a concrete  job  task  or  family  of  job  tasks,  is  an  act  of  personal  insight  and 
judgment.  For  example,  an  innovation  in  instruction  on  aiming  and  firing  hand- 
guns has  perhaps  shown  a 402  efficiency  advantage  over  other  methods  of  instruc- 
tion. Does  the  innovation  also  apply  to  the  aiming  and  firing  of  rifles?  Of 
authomatlc  handheld  weapons?  Of  mortars?  No  generalized  list  can  possibly 
eliminate  the  need  for  some  operational  knowledge  about  the  various  tasks  in- 
volved in  this  decision. 

It  is  realistic  to  assume  that  the  empirical  testing  of  the  innovation  will 
rarely,  if  ever,  sample  from  all  the  potential  range  of  training  problems  to 
which  it  might  apply.  Therefore,  the  potential  value  of  the  innovation  must 
be  supported  by  informed  guesswork.  The  more  "similar"  the  candidate  task  is 
to  the  researched  task,  the  better  the  bet  that  the  candidate  task  will  show  a 
similar  benefit.  But,  what  may  seem  similar  on  external  appearances,  may 
frequently  not  be  similar  psychological ly , and  vice  versa. 

The  following  guidelines  are  therefore  proposed.  The  assessor  has  already 
broadly  classified  the  relevant  job  task  into  one  of  the  five  groupings:  track- 

ing; procedural  ; decision-making;  constructing;  or  leadership-interpersonal. 
Within  this  grouping,  he  will  have  further  classified  the  instructional  objec- 
tive of  the  Innovation  Into  one  or  more  task  structure  elements:  goal-image; 

scan/detect;  Identify;  and  so  on.  He  can  generalize  and  delimit  the  relevant 
range  of  job  tasks  on  one  or  a combination  of  the  concrete  identifying  factors 
that  define  real  tasks  in  real  environments  that  will  be  listed  below. 

The  following  set  of  task  context  variables  should  be  treated  as  coordinate 
variables;  the  data  they  develop  are  a complete  pattern.  In  some  cases,  the 
description  of  the  innovation  will  point  rather  directly  to  the  variable  that 
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is  primary  for  generalizing  range-of-effect.  For  example,  "training  in  use  of 
handheld  soldering  iron  or  soldering  gun",  identifies  a procedure  for  using  a 
tool,  and  all  job  tasks  that  call  for  soldering  iron  skills  are  candidates.  One 
could  search  a data  base  of  descriptions  of  Navy  jobs  directly  with  just  that 
information.  But  an  innovation  claiming  efficient  instruction  in  "strategies 
for  operating  vehicles  through  turbulence"  could,  potentially,  apply  not  only 
to  aircraft  (on  which  the  innovation  may  have  been  validated),  but  also  to 
surface  and  subsurface  vehicles,  as  well  as  land  vehicles  operated  over  rough 
terrain.  In  this  example,  the  key  factor  in  stipulating  a potential  range-of- 
effect  is  "operating  a vehicle  through  turbulence",  so  the  search  should  look 
for  the  operations  and  the  vehicles  (equipments)  operated  in  an  environment  of 
"turbulence".  The  linking  term  is  turbulence. 

These  examples  should  demonstrate  why  it  is  not  practicable  to  set  up  a limited 
set  of  taxonomic  descriptors  with  universal  range  and  applicability  to  job 
training  and  performance. 

The  following  variables,  taken  together,  comprise  a job  task  context  that  must 
necessarily  be  regarded  as  guides,  both  in  probing  for  the  potential  full- 
range  of  the  innovation,  as  well  as  for  delimiting  that  range.  Nor  should  the 
order  in  which  the  variables  appear  determine  the  order  in  which  descriptive 
terms  are  sought;  the  order  should  be  based  on  the  key  characterizations  in  the 
description  and  understanding  of  the  innovation. 

The  objective  variables  that  characterize  task  contexts  are: 

Equipment  and  objects  used  in  the  task 
Environments  in  which  the  task  is  performed 
Tools  used  in  connection  with  the  task 
Reference  information  used  in  performing  the  task 
Task  operations  (made  up  of  "task  structure  elements") 

Criteria  applied  to  task  performance  such  as  "time"  and  kinds  of  "errors". 

PROCEDURE 

1.  If  applicable,  classify  the  training  objective  in  terms  of  target  job  tasks 
into  one  or  more  of  the  following  categories: 

Operations 

Maintenance 

Service 

Corranand. 

This  is  a broad  grouping,  but  It  may  help  as  a rough  first  pass  at  subsetting 
the  inventory  of  Navy  jobs  into  a taraet  group.  It  should  be  noted  that  an 
efficient  search  strategy  is  one  whicn,  as  early  as  possible,  excludes  the 
largest  segment  of  the  total  universe  to  be  searched  as  irrelevant  to  further 
inquiry.  This  is  the  Twenty  Questions  Game  principle.  If  the  innovation 
cuts  across  any  kind  of  categorization  such  as  suggested  in  this  step, 
abandon  it.  In  any  event,  go  on  to  step  2. 
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2.  Using  the  guiding  categories  that  are  named  and  described  in  the  topical 
sections  that  follow,  identify  and  set  down  the  objective  or  external 
characterization  of  the  skills  and  task  structure  elements  that  limit  the 
applicability  of  the  innovation.  For  example,  the  use  of  an  electric  arc 
welder  is  applicable  to  all  jobs  that  would  be  characterized  as:  Pro- 

cedure --  Use  of  Arc  Welder.  But  a turn  on  procedure  specific  to  an  ABA-10X 
equipment  would  clearly  be  restricted  to  the  operator  of  the  ABA-10X  equip- 
ment and  the  field  maintenance  job  on  the  ABA-10X  equipment. 

If  there  is  some  doubt  about  whether  to  use  a more  restrictive  or  a less 
restrictive  characterization,  use  the  less  restrictive,  or  more  general 
characterization.  What  is  now  being  constructed  is  a search  of  candidates 
for  judgments  of  relevance.  The  assessor  will  make  another  examination  of 
the  job  task  descriptions  in  their  individual  contexts,  so  he  will  have  an 
opportunity  to  reject  those  that  do  not  apply.  On  the  other  hand,  if  the 
search  specification  is  more  restrictive  than  it  should  be,  candidates  that 
are  genuinely  relevant  to  the  innovation  will  be  overlooked  in  the  retrieval 
search.  Better  to  cast  the  net  too  wide  than  too  narrow. 

Similarities  for  training  purposes  may  be  established  on  the  basis  of 
patterns  of  similarities  among  the  following  characteristics: 

• Equipment  and  Other  Operational  Objects  Used 

The  name  of  the  equipment  or  class  of  equipment  operated  by  the  operator, 
or  maintained  by  maintenance  personnel,  is  usually  a primary  descriptor 
for  identifying  jobs  and,  by  implication,  of  the  tasks  associated  with 
those  jobs.  If  the  description  of  the  innovation  can  include  relevant 
equipment  names,  the  data  base  search  can  be  narrowed  rather  quickly. 

On  the  other  hand,  if  the  innovation  extends  to  equipment  in  addition 
to  that  initially  specified,  the  greatest  penalty  in  restricting  the 
proper  range  of  the  innovation  will  be  imposed. 

In  assessment  practice,  it  will  be  good  policy  to  identify  equipment 
names  and  types  after  task  structure  elements  and  relevant  environments 
have  been  identifiedV  unless  the  innovation  has  to  do  with  manipulative 
operations.  Manipulative  activities  tend  to  be  equipment  and  tool 
specific.  Note  that  this  recommendation  is  embedded  in  the  present  pro- 
cedure --  see  Subtask  05.13  on  Task  Structure  Elements. 

• Envi ronments 

An  environment  sets  conditions  under  which  a task  is  to  be  performed. 

The  differences  in  conditions  may  imply  a different  kind  or  level  of 
skill.  Aiming  a gun  from  a stable  platform  is  different  from  aiming  a 
gun  from  a moving  platform.  Driving  a vehicle  on  flat  terrain  differs 
from  driving  on  mountainous  terrain,  and  both  differ  from  driving  in 
congested  downtown  traffic.  Performance  in  forgiving  environments. 
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such  as  stall  recovery  at  a high  altitude,  may  require  a skill  that 
differs  both  qualitatively  and  quantitatively  from  performance  of  the 
same  task  in  an  unforgiving  environment,  such  as  stall  recovery  at  a 
very  low  altitude. 

• Tools 

Tools  may  be  regarded  as  a special  kind  of  equipment.  The  ability  to 
use  a given  tool,  such  as  a recording  voltmeter  or  a torque  wrench, 
implies  some  general  range  of  tasks  and  skills,  but  must  usually  be 
qualified  by  the  equipment  or  materials  or  other  work  objects  appli- 
cable to  the  task  context.  It  is  possible,  however,  for  the  name  of 
a tool  in  itself,  to  constitute  a search  specification  to  a job  task 
data  base. 

• Reference  Information 

Reference  information  data  that  may  be  in  the  incumbent’s  head,  or 
available  externally  (such  as  manuals  of  instruction  or  handbooks) 
in  normal,  expected  conditions  of  performing  the  task  which  may  be 
essential  to  carrying  out  the  task.  Examples  are  the  normal  waveshape 
for  a given  electronic  checkpoint,  the  number  of  U.S.  gallons  to  an 
imperial  gallon  to  an  American  motorist  in  Great  Britain,  the  tempera- 
ture at  which  a given  flux  will  bond  given  materials  together  in  the 
task  of  welding. 

Reference  information  may  also  consist  of  procedure  information  to 
supplement  or  substitute  for  memorized  procedures.  Such  supplements 
may  be  necessary  for  procedures  that  are  performed  relatively  rarely 
by  the  incumbent. 

• Task  Operations 

Responses  made  by  the  incumbent  to  task  situations  in  carrying  out  his 
duties  are  denoted  here.  The  description  of  task  operations  may  be 
made  at  many  levels.  Task  actions  can  be  described  in  operational  terms 
with  task  Structure  Elements  as  a general  format  for  scenarios  of  pro- 
cedures, decision-making,  constructions,  and  so  on. 

Task  characterizations  are  generally  weakest  in  the  descriptions  of 
perceptual-cognitive  demands  on  the  incumbent.  However,  these  demands 
are  often  the  most  important  for  skill  learning  and  for  the  transfer  of 
skill  from  one  work  instance  or  situation  to  another.  Scenarios  of 
task  operations  frequently  tend  to  emphasize  the  action  demand  of  the 
moment,  while  neglecting  the  concurrent  range  of  perceptual-cognitive 
activities  that  differentiate  the  mechanical  step-by-step  prescriptions 
For  action  from  the  full  context  of  action  required  for  competence. 

Thus,  safety  and  efficiency  in  driving  in  heavy  traffic  require  not 
only  steering  the  vehicle  in  order  to  avoid  collisions,  but  continuous 
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awareness  of  the  dynamic  changes  in  the  size  of  the  "slot"  in  the 
traffic  which  constrains  the  driver's  maneuverability. 

• Task  Criteria 

These  are  the  variables  in  performance  which  apply  to  determinations 
as  to  whether  the  task  is  performed  effectively  or  ineffectively,  and 
at  what  level  of  effectiveness.  Some  ultimate  criteria  may  be  imposed 
by  nature,  such  as  in  the  recovery  from  a spin  in  an  aircraft  or  by 
the  enemy  in  a combat  situation.  Secondary  criteria  may  be  set  up  by 
instructors  and  evaluation  experts  who  observe  the  extent  to  which  the 
actual  operations  conform  to  prescribed  operations. 

General  criterion  variables  consist  of  time  to  perform  tasks  or  task 
actions  and  kind  and  frequency  of  errors  or  deviations  from  prescribed 
norms.  A third  criterion,  less  explicitly  applied,  may  be  the  quali- 
tative and  quantitative  range  of  situations  with  which  the  incumbent 
may  be  stressed  before  performance  breakdown  or  failure.  Increases  in 
situational  complexity  would  be  an  example  of  such  stress. 

Add  the  job  task  characterizations  derived  from  steps  1 and  2 to  those 
obtained  from  Subtask  05.13*  Select  Task  Structure  Element  or  Task  "Function". 

Organize  the  descriptors  into  one  or  a set  of  search  arguments  with  appro- 
priate logical  connectors.  A logical  connector  is  either  an  AND  or  an  OR, 
or  a NOT. 

For  example: 

Type  of  skill  training:  Construction 

Task  structure  element:  Procedure/Manipulation 

Task  characterized  by: 

Environment:  Improvised  in  field 

Objects  used:  Softwood  OR  hardwood,  NOT  metal 

Tools:  Hand  saws  AND  improvised  bench 

Operation:  Sawing 

Criteria:  Not  applicable. 

(The  example  is  artificially  simplified.) 

Search  the  Navy  inventory  of  job  task  descriptions  for  those  relevant  to 
the  target  knowledge  and  skills  in  the  innovation.  Step  4,  just  preceding 
this  step,  will  have  yielded  one  or  a group  of  search  arguments,  including 
descriptors  and  logical  connectors  between  descriptors.  It  is  expected 
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that  the  descriptors  of  jobs  and  job  tasks  in  the  Navy  inventory  will  have 
matching  descriptors  for  automatic  search.  If  this  is  not  the  case,  human 
interpretations  will  be  required  for  comparing  the  innovation  descriptors 
with  the  job  inventory  descriptors  and  deciding  whether  the  concepts  be- 
hind the  respective  words  used  are,  for  training  purposes,  a match  or  a 
mismatch. 

Bear  in  mind  that  the  final  decision  as  to  whether  a given  job  task  in  the 
Navy  repertory  is  or  is  not  relevant  to  the  innovation,  will  be  an  inspec- 
tion of  the  context  and  tests  of  the  meaning  of  the  descriptors  of  the  Navy 
job  task.  That  is,  the  hits  made  by  the  search  operation  will  not  be 
automatically  accepted  as  relevant,  but  only  as  candidates  for  acceptance. 
This  is  similar  to  making  a search  request  for  a book  in  a library;  the 
search  by  the  librarian  results  In  a number  of  books,  each  of  which  is 
examined  before  making  a final  decision  to  accept  or  reject  the  candidate. 

Search  strategy  has  trade-offs.  A small  set  of  relatively  abstract  descrip- 
tors will  tend  to  result  In  a large  number  of  hits  or  matches  with  items 
in  the  listings,  but  a large  proportion  of  these  hits,  upon  fuller  examina- 
tion of  their  content,  may  turn  out  to  be  irrelevant.  However,  the  proba- 
bility of  getting  a large  proportion  of  relevant  items  will  be  high.  On 
the  other  hand,  a large  number  of  specific  and  qualifying  descriptors  will 
tend  to  result  In  a smaller  number  of  hits  in  the  listing,  and  a larger 
proportion  of  these  hits  will  turn  out  to  be  relevant.  However,  the  risk 
is  increased  that  many  potentially  relevant  candidates  will  be  missed  in 
the  search. 

In  common  practice,  important  searches  are  made  iteratively.  The  first 
net  is  cast  widely  and  its  contents  sampled.  If  the  sample  reveals  a 
large  proportion  of  rejects  on  the  basis  of  judged  irrelevance,  additional 
qualifying  descriptors  are  added  to  the  search  specification  and  the  search 
is  rerun  with  a consequently  smaller  number  of  hits. 

Much  the  same  search  principles  may  be  applied  when  inquiry  is  made  into 
the  knowledge  of  the  expert  on  Navy  job  tasks.  He  will  formulate  an  "image" 
of  the  job  task  characterization  of  relevance  and  evoke  images  from  his 
memory  that  tend  to  match.  The  search  may  be  facilitated  if  the  searcher 
scans  a list  — in  this  case,  of  the  names  of  jobs  and,  perhaps,  duties 
within  jobs  in  the  Navy  inventory  of  jobs  and  duties. 

An  alternative  to  a search  on  job  and  task  listings  and  descriptions  is  to 
search  on  course  names  and  course  objectives.  This  choice  assumes  that  the 
innovation  is  relevant  only  to  those  skills  and  skill  contexts  for  which 
courses  are  available.  This  operation  Is  a jump  to  Subtask  05.16,  Identify 
Relevant  Courses. 

The  result  of  the  search  and  verification  of  job  tasks  relevant  to  the  effect 
of  the  innovation  on  training  and/or  performance  benefits,  is  a list  of  Navy 
jobs  and  job  tasks  (or  skills)  and  such  qualifying  information  as  can  be 
added  to  each  item  on  the  list. 
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Following  the  identification  of  the  Stage  of  Learning  to  which  the  innova- 
tion applies,  the  job  and  task  listing  will  be  used  to  search  and  identify 
relevant  training  courses. 

6.  For  each  job  and  job  task  identified  in  the  Navy  inventory,  obtain  the 
following  information  if  it  is  available: 

• The  number  of  existing  incumbents. 

• Projections  of  the  number  of  students  who  are  to  be  trained  for  the 
job  and  job  task  in  each  of  the  next  five  years. 

• Any  expectations  projected  about  changes  in  the  job  or  job  task 
structure  and  when  they  may  happen  in  the  five  year  future  — as 
these  changes  might  bear  on  the  validity  of  the  innovation  for  the 
job  incumbent. 

• Projections  of  changes  in  the  characteristics  of  the  input  student 
population  to  courses  leading  to  assignment  in  the  jobs  relevant  to 
the  innovation. 

• Projections  of  changes  in  the  conditions  and  environments  for  the 
training  in  the  relevant  jobs  that  could  bear  on  the  practical  value 
of  the  innovation. 

COMMENT:  If  these  kinds  of  data  already  exist  in  the  training  course  descrip- 

tions for  Navy  job  training,  they  need  not  be  obtained  from  the  Navy  job 
inventory  descriptions. 

7.  Determine  if  the  description  of  the  job  in  the  Navy  job  inventory  includes 
identification  of  the  training  courses  prescribed  for  acquiring  the  job 
knowledge  and  skill.  If  so,  take  this  information  and,  if  it  is  sufficiently 
complete  and  up-to-date,  a major  portion  of  Subtask  05.16,  Identify  Relevant 
Courses,  has  been  completed. 

8.  Organize  and  retain  all  data  obtained  in  this  subtask.  It  will  be  most 
useful  if  presented  in  a tabular  format  that  is  keyed  to  the  Navy's  name 
of  job  and  job  task. 


II 1-116 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
05  15  01 


TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  RELEVANT  STAGE  OF  LEARNING 


DESCRIPTION 

Different  stages  of  learning  call  for  different  factors  in  training.  An  innova- 
tion that  may  be  effective  for  one  level  of  skill  acquisition  may  be  irrelevant 
or  even  counter-productive  to  another  level  of  skill.  Differences  in  skill 
level  training  may  be  reflected  in  course  levels,  but  more  specifically  in 
units  of  training  within  a course.  The  characterization  of  skill  level,  there- 
fore, enables  a more  precise  determination  of  the  training  operations  to  which 
the  innovation  is  specifically  applicable. 

The  practical  aspects  of  stages  of  learning  are  those  of  acquiring  a skill. 

The  reason  for  concern  with  stages  of  learning  is  that  the  minimum  fidelity  to 
the  job  task  for  effective  progress  in  skill  acquisition  changes  from  one  stage 
and  level  to  another.  Whereas  words,  diagrams,  and  pictures  may  be  adequate 
and  effective  at  an  early  stage  in  acquiring  a skill,  or  the  components  of 
skilled  performance,  later  stages  demand  more  physical  and  functional  fidelity 
for  useful  growth  and  transfer  of  skill  from  training  to  operations.  And  where- 
as, in  early  phases  of  skill  learning,  adequate  student  response  may  be  verbal, 
in  later  stages  the  student  response  must  be  the  literal  equivalent  of  the  job 
task  response  with  the  physical  work  objects  and  tools  of  the  job.  Temporal 
and  other  dynamic  relationships  must  become  realistic  so  that  the  student 
develops  proper  cause-effect  "anticipations"  of  the  tool,  equipment,  material, 
and  environment.  At  high  stages  of  skill  learning,  the  full  information- 
handling load  of  simulated  or  real  environmental  conditions  and  contingencies 
must  be  presented,  else  the  skill  will  level  off  at  less  than  a reliable,  on- 
the-job  competence.  Not  only  must  higher  degrees  of  physical  fidelity  to  the 
real  job  task  accompany  higher  stages  of  task  learning,  but  the  form  of  measure- 
ment and  analysis  of  student  performance  changes,  as  do  the  desirable  properties 
of  instructional  control. 

Sophisticated  awareness  of  cost-effective  instruction  will  take  into  account  the 
minimum  requirement  for  students,  consistent  with  effective  learning  at  a given 
stage,  and  for  the  transfer  of  that  learning  to  a later  stage  of  what  is  being 
practiced  and  learned.  For  example,  a growing  body  of  studies  is  revealing  the 
fact  that  many  sequential  procedures  can  be  learned  to  near-operational  levels 
of  proficiency  on  relatively  crude  mockups,  rather  than  on  operational  equipment 
or  expensive,  high-fidelity  simulators.  This  section  of  training  analysis, 
therefore,  characterizes  the  major  stages  of  skill  learning  at  which  various 
kinds  of  instructional  options,  with  substantial  implications  for  cost-effect- 
iveness, can  occur. 

A summary  of  stages  of  learning  emphasizes  the  student's  learning  processes. 

With  continued  practice,  the  behavior  processes  within  the  student  tend  to 
change.  These  changes  are  more  or  less  systematic  and  predictable,  and  depend 
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on  instructional  sequencing,  kind  of  presentation  of  content,  and  the  nature 
and  extent  of  conceptual  content.  ("Concept",  here,  refers  to  an  idea  of  a 
property  or  class,  not  merely  the  name  of  a word.)  Transitions  are  gradual 
rather  than  abrupt,  and  for  a given  student  not  all  aspects  of  a task  being 
learned  make  the  same  shift  at  the  same  time.  The  major  transition  is  from 
the  stage  where  the  student  gives  himself  verbal  instructions  about  what  next 
to  notice  or  do,  to  the  stage  where  the  same  behavior  is  largely  automatic  -- 
that  is,  executed  with  little  or  no  verbal  mediation.  When  a class  of  behavior 
has  become  "automated",  the  effective  channel  capacity  of  the  student  (what  he 
can  carry  in  his  head  at  one  time)  becomes  increased  so  that  he  can  process 
more  information  at  the  awareness  level.  The  source  of  the  information  may  be 
additional  kinds  of  task  input  data,  feedback,  or  guidance  from  the  instruc- 
tional vehicle.  This  automatization  of  behavior  tends  to  develop  in  perceptual 
task  activities,  cognitive  activities,  the  organization  of  motor  activities, 
and  in  perceptual -motor  activities. 

What  tends  to  get  automated  are  the  relative  constants  among  a set  of  task 
experiences  in  what  is  perceived,  or  thought  about,  or  the  movement  patterns 
executed,  or  the  relationships  among  all  three.  These  constants,  in  terms  of 
conditions  and  processes,  among  a set  of  task  experiences  may  be  thought  of  as 
a "task  format".  A constant  may  consist  of  a variable  in  a task  input,  pro- 
cessing link,  or  task  output.  An  example  is  "altitude"  in  flying  conceptually, 
or  the  altimeter  in  actually  flying  the  aircraft.  The  momentary  reading  on  the 
altimeter  is  the  value  which  this  variable  may  take.  Other  variables  which  are 
consistently  required  in  heavier-than-ai r flight  are  course  heading,  attitude, 
and  air  speed.  The  information  processing  format  of  necessary  variables 
(called  constants  above)  for  flying  includes,  but  is  not  necessarily  restricted 
to,  this  group.  After  having  internalized  this  format  of  variables  by  extensive 
task  practice,  the  pilot  can  more  readily  use  different  physical  sources  of 
data  (such  as  a different  kind  of  altimeter  dial)  for  given  variables,  than 
the  more  inexperienced  pilot  who  deoends  on  procedural  sequences  for  information. 
A task  format  is  somewhat  analogous  to  a printed  form  that  has  been  standardized 
by  a business  for  a given  kind  of  transaction,  such  as  a bill  or  invoice  to  a 
customer.  Task  formats  also  apply  to  routinized  decisions,  and  to  construc- 
tions such  as  routinized  planning. 

The  concept  of  task  format,  and  the  processes  whereby  task  formats  are  required 
is  essential  to  account  for  (hence,  to  train  for)  the  transfer  of  training  (or 
what  has  been  learned  by  the  student)  from  one  practice  environment  to  another. 

An  attempt  is  made  to  adapt  stage  of  learning  differentiations  to  more  or  less 
fit  the  terms  and  descriptions  in  Navy  training  documentation.  The  translation 
will  by  no  means  be  a precise  fit  to  the  processes  described  above,  but  the 
approximation  may  be  adequate  for  interpreting  the  intent  in  course  descrip- 
tions, course  element  objectives,  instructor  manuals,  and  student  manuals. 

PROCEDURE 

1.  Determine  to  which  one  or  more  of  the  following  stages  of  learning  the 
innovation  is  restricted  in  its  applicability. 
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• Orientation  and  Familiarization 

These  may  be  general  reference  and  context  knowledges.  They  may  include 
information  about  goal  states,  environments,  appearance  and  properties 
of  tools  and  other  resources,  communications  and  job  interfaces,  roles 
on  the  job  and  roles  in  the  training  environment,  and  relationships  of 
tasks  to  each  other.  If  these  kinds  of  information  are  not  presented 
and  credible  to  the  student,  he  will  tend  to  develop  his  own  images 
and  hypotheses.  The  concepts  acquired  during  orientation  may,  in  some 
cases,  serve  the  student  as  mnemonic  models  for  learning,  retention, 
and  generalization  — that  is,  he  acquires  a general  picture  of  what  to 
expect  and  the  more  specific  content  he  learns  gets  tied  in  with  these 
expectations.  The  expectations  can  serve  to  organize  the  details  of 
what  is  learned  later. 

Theory  of  operation  is  included  here  because  it  may  serve  mainly  as 
orientation  to  the  skill,  rather  than  as  a component  of  the  skill.  It 
becomes  a skill  component  only  if  it  becomes  integrated  with  practice 
in  one  or  more  of  the  actual  tasks  performed  by  the  incumbent,  and  the 
"theory"  information  must  be  referenced  by  the  student  (or  job  incum- 
bent), rather  than  direct  procedural  instructions  from  a procedurally 
oriented  job  manual.  But  theory  of  operation  may  apply  to  various  kinds 
of  content;  what  the  system  is  supposed  to  do  and  how  it  does  it  in 
principle,  criteria  of  system  effectiveness,  environments  in  which  the 
system  is  to  operate,  and  interdependence  of  subsystems,  physically  and 
functionally.  Theory  of  operation  may  include  some,  and  possibly  all, 
of  the  nomenclature  associated  with  learning  and  performing  tasks  by 
incumbents. 

The  instructional  vehicle  requirements  are  essentially  those  for  pre- 
senting text,  diagrams,  and  pictures  (static  and  dynamic),  and  for 
receiving  and  evaluating  symbolic  responses  from  the  student. 

• Task  Nomenclature  (Identifications,  Locations,  and  Names)  and  Facts, 
Rules 

This  stage  may  be  merged  into,  or  be  an  extension  of,  the  orientation 
and  familiarization  phase.  The  student  learns  to  identify  and  differ- 
entiate task-related  entities  (situations  and  objects  and  operations) 
and  to  associate  names  with  them.  By  learning  nomenclatures,  the 
student  can  be  briefed  and  symbolically  guided  by  the  instructional 
vehicle.  This  enables  symbolic  rehearsal  of  task  actions.  It  also 
facilitates  the  student's  ability  to  mediate  between  situation  and 
overt  task  response,  by  giving  himself  instructions  with  words  and 
images  when  the  nomenclature  is  organized  into  descriptions  of  task 
procedures. 

The  instructional  vehicle  requires  the  ability  to  present  the  appearance 
or  image  of  the  entity  paired  with  its  label.  This  pairing  of  appear- 
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ance  and  name  may  be  presented  by  a variety  of  means.  In  general, 
identifications  of  things  are  learned  more  quickly  when  they  are 
shown  in  meaningful  contexts. 

Facts,  such  as  the  color  coding  of  resistors  for  their  resistance  values, 
or  the  weight  and  dimensions  of  a given  piece  of  apparatus,  can  be 
learned  with  similar  vehicle  properties. 

The  learning  of  direct  symbol  translation  rules,  such  as  for  convert- 
ing inches  into  meters,  depends  on  minor  if  any  change  in  properties 
to  the  instructional  vehicle  for  facts  and  nomenclatures.  The  proviso 
is  that  the  rule  is  a simple,  one-for-one,  translation  of  elements  in 
one  symbol  set  (such  as  degrees  in  Fahrenheit)  to  elements  in  another 
symbol  set  (degrees  in  Centigrade). 

• Task  Formats  at  the  Conceptual  Level 

At  the  termination  of  this  phase,  the  student  is  able  to  give  a more 
or  less  specific  description  of  what  initiates  the  task  or  duty  being 
studied,  and  of  the  major  sequences  of  activity  that  comprise  the  task. 
Where  operational  task  formats  are  learned,  the  student  should  also  be 
able  to  state  or  identify  the  major  variables  comprising  (a)  the  task 
situation  or  problem,  and  (b)  the  goodness  criteria  in  the  task  outcome. 
In  another  form  of  test,  the  student  is  presented  with  motion  picture 
sequences  of  the  task  being  performed  and  he  describes  what  is  going 
on  and  why. 

This  phase  may  be  more  specific  with  respect  to  job  tasks  than  famil- 
iarization, but  less  specific  than  the  learning  or  memorizing  of  the 
descriptions  of  step-by-step  procedures.  The  Navy  officer  does  not 
usually  learn  how  to  perform  the  actual  task  itself,  but  gets  a 
"working  knowledge"  about  the  performance  of  the  task. 

A task  format  at  the  conceptual  level  is  a general  map  of  what  is  done 
in  disassembling-replacing-assembling  an  electrical  or  mechanical 
assembly,  the  general  pattern  of  operations  in  sighting  a piece  (or 
class)  of  fire  control  equipment,  or  the  general  steps  in  making  a 
chemical  analysis  of  the  water  in  a boiler.  This  conceptual  knowledge 
may  become  a cognitive  "organizer"  for  actually  learning  the  task  as  a 
verbal izable  procedure,  and  actually  performing  it.  Because  the  concept 
of  task  format  is  not  formalized  in  Navy  training  documentation,  this 
is  an  "implicit  category"  at  present.  Its  meaning  as  a training  ob- 
jective can  be  interpreted  in  the  content  of  some  theory  courses. 

The  training  vehicle  presents  symbolic  materials  and  the  student  makes 
symbolic  responses.  In  addition,  the  presentation  may  include  live 
demonstrations  of  task  performance,  or  representations  of  it,  ranging 
from  text  descriptions  through  motion  pictures.  More  sophisticated 
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and  "enriched"  presentations  may  present  the  task  action  as  a "cause", 
and  some  system  outcome  as  a related  "effect".  This  material  is  use- 
ful as  a training  experience  to  the  extent  that  it  is  remembered  and 
transferred  to  relevant  work  situations,  or  to  later  training 
situations. 

Note  that  this  stage  of  learning  is  equivalent  to  the  content  of  instruc- 
tion to  be  specified  in  Subtask  05.22,  Identify  Type  of  Task  Format. 

If  analysis  was  made  in  Subtask  05.22,  its  results  can  be  carried  over 
directly  into  this  topic,  and  possibly  the  next  topic  below.  See  Sub- 
task 05.22  for  a more  analytic  description  of  task  formats. 

• Procedures  Learned  at  the  Verbal  Level  Only 

The  student  demonstrates  his  completion  of  this  phase  when  he  can  either 
describe  in  detail  the  steps  in  the  task  procedures,  including  the  signs 
or  cues  that  specify  the  task  response  to  make,  or  preferably  when  he 
can,  in  the  presence  of  the  task  objects  and  environments,  describe 
verbally,  and  by  pointing,  each  step  in  the  task  procedure. 

It  is  possible  to  memorize  the  verbal  equivalent  of  a set  of  printed 
job  instructions.  The  evaluation  of  incumbents  on  their  proficiency  in 
the  training  path  system  used  in  submarine  and  fire  control  contexts, 
puts  heavy  emphasis  on  the  ability  of  the  incumbent  to  give  verbal 
descriptions  of  procedures  to  be  followed  in  the  array  of  tasks  making 
up  a requirements  pattern.  The  verbalized  procedure  may  serve  as  a more 
effective  precursor  to  learning  the  live  context  procedure  if  the 
student  can  readily  imagine  the  appearance  of  the  presented  task  situa- 
tions and  the  overt  task  operations  as  the  incumbent  performs  them.  This 
implies  effective  learning  of  "nomenclature  --  identification  and  loca- 
tions and  names"  and  their  presentation  in  the  context  of  rehearsing  or 
practicing  the  steps  in  the  verbal  procedure. 

Devices  more  sophisticated  than  printed  text  can  increase  the  efficiency 
of  learning  the  verbalized  procedure,  its  reliability,  and  especially 
its  transferability  to  the  actual  or  simulated  physical  work  situation 
and  actual  procedural  performance.  But  the  display  and  student  response 
acceptance  facilities  of  even  this  device  may  be  symbolic  representations 
rather  than  physical  representations. 

• Performing  Task  Components  with  Guidance 

This  level  corresponds  to  actually  carrying  out  the  task  in  full,  but 
with  some  degree  of  external  prompting. 

In  Navy  language,  a skill  consists  of  a "doing  operation"  --  that  is, 
actual  task  behaviors.  Performing  a task  with  guidance  is  equivalent  to 
a "middle  stage  of  learning"  --  where  the  student  is  giving  himself  verbal 
instructions  in  task  performance  (mediated  performance)  with  growing 
degrees  of  independence  from  outside  sources  of  task  information. 
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• Doing  the  Entire  Task/ Job  Procedural ly:  Barely  Acceptable  Mastery. 

The  student  or  incumbent  is  able  to  make  his  way  through  the  task  un- 
aided, but  with  errors  and  fumbling  and  at  a slow,  uneven  pace;  he  is 
unable  to  attend  to  other  tasks  or  ongoing  activities  at  the  same  time. 
This  should  be  .the  equivalent  of  a late  stage  in  "middle  learning",  where 
subtasks  are  getting  organized  into  performance,  but  where  getting 
through  the  procedures  correctly  uses  up  most  of  the  student’s  attention 
capacity.  The  range  of  his  job  experiences  has  not  yet  been  great 
enough  to  give  him  a wide  repertory  of  response  patterns  for  contin- 
gencies, nor  has  he  had  enough  practice  in  doing  the  task  to  make  its 
behavioral  elements  more  or  less  automatic,  like  subroutines.  This 
category  can  have  its  equivalent  in  the  Navy  evaluation  expression  of 
"performs  task  procedures  without  supervision",  if  the  expression 
implies  "meets  minimum  or  marginal  acceptable  standards  for  job  per 
formance". 

This  level  may  be  reached  after  some  hours  of  procedural  practice  on 
the  raw  equipment,  with  the  help  of  manuals  of  instruction  and/or  the 
divided  attention  of  an  instructor  as  guide.  The  performance  level 
deteriorates  rapidly  during  a few  days  of  nonpractice  and  is  not 
rapidly  restored  with  renewed  practice. 

• Highly  Proficient  Task/ Job  Performance  in  Work  Context 

The  student  or  job  incumbent  performs  the  entire  task  without  hesita- 
tions in  a manner  that  indicates  that  he  is  anticipating  and  making 
provision  for  future  steps  as  he  proceeds;  he  is  also  able  to  attend 
to  and  respond  to  other  concurrent  activities. 

This  is  a learning  phase  and  a skill  level  not  identified  in  most  course 
objectives,  nor  in  incumbent  evaluations.  As  a stage  in  learning,  it  is 
characterized  by  a high  degree  of  "automatized"  performance  of  routinized 
task  operations,  including  components  of  many  decision-making  operations 
which  now  are  performed  with  comparatively  little  mental  "figuring  out 
the  solution".  At  this  level,  the  student  or  incumbent  can  perform  a 
task  strategically;  he  can  optimize  resources  across  a number  of  steps 
rather  than  proceeding  merely  one  step  at  a time.  Not  only  can  he 
plan  several  steps  ahead,  he  can  anticipate  conditions  and  situations, 
perform  efficiently  as  well  as  effectively,  and  keep  in  mind  special 
information  about  the  environment,  the  mission,  priorities,  and  other 
specifics  that  he  could  not  do  if  his  attention  was  completely  dominated 
by  trying  to  remember  what  next  to  do.  This  level  implies  that  many 
procedural  operations  (whether  correct  or  incorrect)  have  been  learned 
so  well  that  they  are  virtually  automatic,  and  that  his  anticipations  are 
effective  in  transitioning  his  activity,  smoothly,  from  one  sequence  of 
responses  to  another.  This  level  of  skill  is  well -retained  over  long 
periods  of  time  and  is  quickly  relearned  from  any  degradation  in  recall. 
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This  stage  is  reached  only  after  extensive  and  varied  repetitive 
practice  that  makes  the  full  demands  on  attention  required  in  the  job. 
But  it  can  be  erroneous  to  assume  that  only  job  experiences  in  great 
number  and  variety  can  lead  to  this  stage.  We  should  recognize  that 
job  experience  is  also  a learning  experience,  and  any  learning  expe- 
rience is  potentially  subject  to  deliberately  induced  training  actions. 
Wrong  procedural  responses  can  be  overlearned  as  readily  as  correct 
procedural  responses.  It  is  a good  working  hypothesis  that  many  job 
incumbents- who  could  reach  the  level  that  combines  smooth,  reliable 
automatic  response  with  correctness  of  response  and  strategic  capa- 
bilities, fail  to  do  so  for  reasons  that  could  have  been  controlled 
by  training  type  decisions  and  actions,  and  that  many  others  are  de- 
layed from  reaching  this  level  for  the  same  reason. 

Rather  high  fidelity  of  performance  requirements,  both  in  kind  and 
range,  of  task  situations  is  required,  plus  instructional  functions 
of  response  evaluation  and  immediate  corrective  feedback.  Plus  guid- 
ance functions.  Plus  very  extensive  blocks  of  time  for  practice. 

• Unusual  Task  Conditions 

This  is  learning  to  cope  with  the  multitude  of  things  that  can  go  wrong 
such  as  from  human  error,  equipment  malfunction,  extraordinary  environ- 
mental conditions,  incorrect  information,  and  so  forth.  This  is  supple 
mental  training  rather  than  a stage  in  learning,  although  it  may  be 
introduced  in  a stage  of  learning  regimen.  The  learner  should  be 
competent  In  handling  routine  operations  before  having  the  additional 
learning  burden  of  exceptional  conditions  to  cope  with.  Unless  the 
job  incumbent  can  cope  with  exceptional  conditions  peculiar  to  the  job 
task,  his  practical  usefulness  is  limited. 

Because  of  the  rare  opportunities  for  the  incumbent  to  practice  rare 
events  on  the  job,  training  for  exceptional  conditions  — such  as 
emergencies  --  requires  large  redundancies  in  the  content  of  what  is 
learned,  in  order  to  achieve  a useful  reliability  in  recall  when  the 
situation  demands  recall.  Retention  of  procedural  skills  over  long 
periods  of  disuse  will  also  be  supported  by  periodic  refresher  type 
training.  This  is  likely  to  be  most  effective  if  given  in  the  full 
context  of  task  performance,  such  as  is  the  case  with  refresher  train- 
ing in  emergencies  for  aircrew  given  in  simulators.  But  any  operators 
or  maintenance  personnel  are  faced  with  unusual  task  conditions  which 
challenge  mission  accomplishment. 

• Performing  Job/Tasks  at  Key  Man  Level 

This  is  demonstrated  by  performing  the  job  at  a level  of  skill  that  is 
recognized  by  others  as  unusually  proficient  so  that  the  individual  is 
depended  on  to  contribute  his  expertise  in  situations  when  others 
having  the  same  job  are  by-passed  or  have  failed. 
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There  can  be  a combination  of  training,  job  experience,  aptitude  and 
motivations  that  make  some  individuals  the  key  technical  men  in  their 
organization  and  specialty.  These  individuals  tend  to  contribute  far 
more  than  their  share  to  the  mission  effectiveness  levels  of  their 
organization.  The  key  person  phenomenon  is  likely  to  be  more  than  a 
mere  sum  of  skilled  performances,  hence  applicable  to  the  person  rather 
than  merely  his  skill  attributes.  In  some  cases,  the  key  person  may 
not  have  done  well  in  academic  phases  of  training  such  as  often 
demanded  by  the  conceptual  demands  of  theory  of  operation  or  verbal 
memorization  of  procedures  and  other  rote  demands.  But  the  practical 
importance  of  such  individuals  in  organizations  makes  it  desirable 
to  identify  and  characterize  both  the  role  and  the  skill  level  in  any 
complete  range  of  description  of  competence. 

The  development  of  the  key  man  probably  requires  substantial  catering 
to  individual  differences  and  more  or  less  unique  training  routes. 

• Refresher  Learning 

At  the  beginning  of  refresher  training  the  student  cannot  perform  or 
recall  to  criterion  performance  levels,  tasks  and  task  information 
which  he  was  once  able  to  perform,  but  at  the  conclusion  of  refresher 
training  he  is  again  able  to  perform  at  his  once  established  level  or 
better. 

This  is  a "stage"  of  learning  intended  to  compensate  for  the  forgetting 
effects  of  extended  periods  of  time  with  no  practice  of  the  skill,  and/ 
or  intervening  experiences  some  of  which  may  interfere  with  the  recall 
of  the  correct  responses  to  make  in  the  old  task. 

If  the  tasks  were  learned  to  high  levels  and  performed  a large  number 
of  times,  recall  with  practice  may  be  rapid.  But  the  rapidity  of  re- 
call and  full  reinstatement  of  the  original  skill  may  be  slowed  or  in- 
creased depending  on  conditions  of  refresher  practice.  The  principle 
of  effective  retraining  of  an  old  skill  is  that  of  inducing  any  other 
content  of  retention:  the  greater  the  extent  to  which  the  original 

situation  and  all  of  its  psychological  context  is  reproduced,  the 
greater  the  probability  of  recall  of  the  responses  made  to  the  original 
situation.  In  refresher  training,  a highly  significant  factor  in  that 
context  is  the  actual  making  of  the  responses  (cognitive  and  motor)  under 
the  same  conditions  of  response  made  in  the  original  learning.  This  is 
not  altogether  a paradox. 

Take  an  example.  You  once  memorized  a poem.  Years  later  you  are  given 
the  title  and  asked  to  speak  the  poem.  Not  a single  word  can  be  re- 
called. A prompter  gives  you  the  opening  phrase.  You  speak  the  opening 
phrase  aloud.  The  probability  of  your  being  able  to  continue  the  words 
of  the  poem  vastly  increases.  As  you  complete  the  utterance  of  the 
first  line  of  the  poem,  the  probability  of  your  recall  of  the  second 
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line  increases  still  more.  If  all  sections  of  the  poem  were  learned 
equally  well,  each  additional  phrase  you  speak  continues  to  increase 
the  probability  of  remembering  the  remainder  of  the  poem.  If  you 
learned  it  by  speaking  it  aloud,  it  will  best  be  recalled  by  speaking 
It  aloud.  The  recall  is  assisted  by  the  "response  context"  which, 
especially  in  serial  activities,  becomes  the  most  significant  set  of 
cues  for  recall.  But  the  effectiveness  of  response  context  is  not 
limited  to  serial  activities. 

This  principle  helps  to  understand  that  attempting  to  relearn  the 
skill  by  practice  which  simulates  the  original  early  learning  conditions 
will  be  highly  inefficient  for  redintegrating  the  old  skill  at  its 
terminal  level  when  it  ceased  to  be  practiced.  The  kinds  of  responses 
made  in  early  practice  are  not  the  kinds  of  responses  made  at  full 
competence  level.  Relearning  an  old  skill  should  attempt  to  simulate 
the  terminal  levels  of  performance  of  the  skill.  When  its  psychological 
context  has  been  restored,  practice  may  concentrate  on  ironing  out  the 
deficiencies. 

Studies  have  shown  that  procedural  skills,  such  as  emergency  operations 
in  an  aircraft,  if  once  learned  to  a high  level  of  performance  reli- 
ability, can  be  "refreshed"  as  effectively  by  verbal  rehearsal  as  by 
practice  in  the  simulator  of  the  actual  procedures.  No  doubt  the  verbal 
rehearsal,  to  be  effective,  must  be  accompanied  by  some  internal  imagina- 
tion of  the  cues  and  responses  connoted  by  the  verbalizations.  It  is 
unlikely  that  procedural  skills  learned  only  to  intermediate  levels  can 
be  effectively  refreshed  by  symbolic  rehearsal.  Skills  dependent  on 
complex  perceptual -motor  or  motor  patterns  will  probably  have  little  or 
no  refresher  value  derived  from  verbal  rehearsal,  except  perhaps  to 
alert  the  operator  to  a changed  context  of  cues.  Example,  the  American 
driver  who  has  returned  from  driving  in  the  United  Kingdom,  when  getting 
into  his  car  at  the  airport  reminding  himself  that  in  the  U.S.A.  one 
drives  on  the  right  rather  than  the  left  of  the  road,  and  rehearses  some 
mental  imagery  to  that  effect. 

2.  Add  this  Stage  of  Learning  characterization  to  the  description  of  job  task 
skills  that  emerged  from  the  determination  of  relevant  job  tasks  in  the 
Navy  inventory  of  jobs.  Subtask  05.14.  The  assessor  can  now  formulate  a 
search  argument  for  retrieval  of  rel event  courses  from  the  course  descrip- 
tion data  base. 

COMMENT 

The  foregoing  definitions  of  learning  and  performance  levels  are  intended  to 
help  assign  the  relevance  of  an  innovation  in  training  by  persons  with  some 
background  in  the  psychology  of  learning  and  training.  Existing  Navy  terminology 
and  distinctions  were  adapted  where  possible  in  this  determination  of  relevance. 
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but  not  all  concepts  of  learning  level  described  here  are  represented  in  formal 
Navy  documents  about  course  objectives  on  one  hand,  or  performance  criteria  on 
the  other.  The  possibility  should  exist  that  Navy  practices  may  be  at  least 
moderately  changed  to  adapt  to  the  distinctions  described  here,  and  for  more 
reasons  than  merely  to  evaluate  innovations  in  training  technology.  For 
example,  the  least  overall  cost-effective  level  of  learning,  followed  by  a 
period  of  disuse  after  which  on-the-job  retention  and  transfer  is  measured, 
is  likely  to  be  the  "level  of  bare  procedural  mastery"  which  terminates  much 
formal  training. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 

SUBTASK:  IDENTIFY  RELEVANT  COURSES 


DESCRIPTION 

At  this  point  in  the  task  content  skill  route,  the  following  have  been  iden- 
tified: 

The  general  class  of  skill 

The  task  structure  elements  that  are  applicable 

The  relevant  job  tasks  from  the  Navy  inventory 

The  applicable  stage(s)  of  learning,  including  formats  for  procedures, 
decisions,  and  constructions. 

This  information  should  enable  a variety  of  search  strategies  to  be  applied 
to  a data  base  of  course  and  course  objectives  descriptions. 

As  in  search  operations  described  previously,  the  two  stage  activity  consists 
of  using  the  search  specification  to  get  a list  of  candidate  courses  poten- 
tially relevent  to  the  innovation,  and  then  validating  or  rejecting  each 
candidate  on  the  basis  of  informed  judgment  based  on  at  least  a general  know- 
ledge of  the  course  content.  Mere  descriptive  labels  of  courses  and  course 
objectives  will  generally  be  insufficient  for  the  validation  by  judgment  phase. 

Because  many  courses  are  set  up  to  fill  standard  time  slots  in  terms  of  weeks, 
there  is  bound  to  be  arbitrariness  as  to  the  stage  or  level  of  a skill  that  a 
given  course  will  carry  a body  of  students.  This  fact  justifies  a "stage  of 
Skill  Learning"  characterization  applied  to  the  innovation. 

The  identification  of  courses  in  the  curriculum  relevant  to  the  innovation 
may  also  have  to  depend  on  the  names  and  objectives  of  training  units  within 
the  courses.  Unfortunately,  if  there  is  not  a match  between  the  nomenclature 
used  in  the  search  specification  and  nomenclature  used  to  describe  the  train- 
ing units  and  their  objectives,  a knowledgeable  person  may  have  to  mediate 
the  translation. 

The  Identification  of  training  units  within  courses  will  have  to  be  made  in 
the  next  step,  if  it  has  not  been  done  here. 

PROCEDURE 

1.  Identify  by  title  and  other  Identifying  code,  each  course  of  instruction, 
current  or  projected,  that  fits  the  criteria  of  relevance  to  the  innovation. 
Each  course  so  identified  will  presumably  contain  one  or  more  training  units 
or  elements  (or  portions  of  the  course)  specifically  relevant  to  the  innova- 
tion. 
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2.  Even  those  innovations  intended  only  for  on-the-job  benefits  should  have 
a determination  of  the  relevant  courses  so  that  a training  cost  reference 
may  be  established. 

3.  The  relevant  training  courses  should  be  unambiguously  identified  so  that 
the  cost  analysis  performed  in  later  tasks  can  find  the  correct  reference 
data. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFY  RELEVANT  TRAINING  UNITS  WITHIN  COURSES 


DESCRIPTION 

Much  or  all  of  this  subtask  may  have  been  accomplished  in  the  process  of 
identification  of  relevant  courses,  Subtask  05.16.  The  determination  of 
training  unit  applicability  may  have  been  somewhat  perfunctory.  Furthermore, 
the  identification  of  merely  one  relevant  training  unit  within  the  course  would 
justify  including  the  course  as  relevant,  without  examination  of  the  possibility 
of  additional  training  units  also  being  applicable. 

The  present  sub task,  therefore,  requires  an  examination  of  all  of  the  constit- 
uent training  units  and  objectives  with  a course  of  Instruction  for  meeting 
the  contextual,  as  well  as  descriptor,  matching  of  innovation  to  training  re- 
quirement and  training  process.  All  previous  analytic  descriptor  selection 
and  search  operations  have  focussed  forward  to  the  decisions  of  acceptance  or 
rejection  for  relevance  to  be  made  in  this  subtask.  That  is  to  say,  only 
candidates  for  acceptance  have  been  selected. 

PROCEDURE 

1.  Review  carefully  the  description  of  the  Innovation  in  Task  1.  Note  not  only 
its  targeted  objectives,  but  the  contexts  In  which  the  empirical  data  were 
developed,  and  other  substantive  aspects  of  the  innovation.  This  context  of 
information  should  be  held  In  mind  while  examining  the  context  of  the 
descriptions  of  each  candidate  training  unit  within  the  course. 

2.  Examine  the  context  of  the  potentially  applicable  training  unit  with  equal 
care.  A guide  for  comparison  is  not  so  much  in  the  training  operation  as 
In  the  clear  identification  of  the  specific  skill  or  skill  component  that 
is  to  be  learned  by  the  student.  This  is  an  operational  reference  which 
is  generally  better  than  merely  a verbal  reference.  The  judgment  of  rele- 
vance may  be  facilitated  by  anticipating  the  next  subtask  by  asking,  "Will 
the  innovation  really  Improve,  or  not  improve  and  impede  the  skill  element 
that  Is  being  learned  here?"  If  a strong  doubt  exists,  it  may  be  better  to 
reject  than  to  accept  a candidate,  so  as  to  avoid  unduly  inflating  the 
estimate  of  potential  benefit  from  the  innovation. 

3.  If  the  innovation  aims  at  skill  components  that  are  clearly  not  identified 
In  otherwise  clearly  relevant  courses,  these  components  must  be  so  identi- 
fied so  that  they  can  be  evaluated  as  on-the-job  benefits.  The  reference 
course  should  be  identified,  and  the  training  units  or  objectives  most 
appropriate  by  the  acquisition  of  the  added  skill  identified.  This  will 
enable  a cost  analysis  for  adding  the  necessary  training  to  be  balanced 
against  assessment  of  on-the-job  value. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  ASSESS  THE  TRAINING  BENEFIT  PATTERN 


DESCRIPTION 

The  assessment  procedure  on  the  task  content  route  has,  to  this  point,  identi- 
fied the  courses  and  training  units  within  courses  that  are  relevant  to  the 
innovation.  In  this  step,  the  magnitude  of  the  benefit  will  be  assessed,  the 
importance  of  the  kind  and  magnitude  of  the  benefit  weighted,  and  the  proba- 
bility of  achieving  the  benefit  estimated. 

The  major  target  types  of  benefit  consist  of:  training  time  reduced,  aptitude 

level  reduced,  attrition  level  reduced,  and  performance  level  increased  or 
extended.  The  benefit  targets  that  can  be  most  readily  translated  into  cost 
advantages  are  reduction  in  training  time  and  reductions  in  attrition  levels. 

Note  that  when  relevant  courses  have  been  identified,  the  cost  assessment  model 
will  identify  the  number  of  students  and  other  multiples  of  the  unit  of  benefit 
that  have  been  estimated  for  the  course. 

PROCEDURE 

1.  Answer  questions  about  reduction  in  training  time. 

Each  of  the  following  questions  should  be  applied  to  each  of  the  training 
course  elements  that  is  potentially  relevant  to  the  innovation.  If  more 
than  one  judge  makes  independent  estimates  about  the  same  entity,  the 
evaluation  program  will  average  the  estimates,  but  also  use  the  distribu- 
tion of  judgments  as  a measure  of  reliability  of  judgment  about  the  effects 
of  the  innovation  on  that  training  course  element. 

Question  1A.  For  each  training  element  (or  unit)  relevant  to  the  innova- 
tion, what  is  the  estimated  percentage  reduction  in  training  time  for  that 
element?  K * a 


The  estimate  will  be  a judgmental  extrapolation  from  the  description  of 
the  innovation  made  in  Task  1,  and  any  empirical  data  contained  in  that 
description,  modified  by  subsequent  investigation  by  the  assessor.  This 
estimate  assumes  that  the  innovation  is  technically  feasible  and  is  applied 
in  moderately  favorable  conditions. 

If,  however,  the  assessor  believes  that  if  the  innovation  is  introduced, 
training  time  will  actually  increase  rather  than  decrease  or  remain  as  it 
is,  he  should  go  to  Question  1A  Negative. 
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Question  1A  Negative 

Assume  the  assessor  believes  the  innovation  will  increase  training  time 
over  comparable  conditions.  He  would  specify  NEGATIVE  and  the  percentage 
of  the  training  time  increased  over  the  standard  or  projected  training 
time  for  the  training  element.  He  would  then  answer  Questions  IB,  1C,  and 
ID  as  they  are  put,  but  with  reference  to  the  negative  value. 

Question  IB.  For  each  of  the  next  five  years,  what  is  the  number  of 
students  that  will  be  affected  in  that  training  element  or  unit? 

Question  1C.  What  is  the  relative  importance  of  this  particular  benefit 
in  relation  to  all  of  the  other  benefits  (or  liabilities)  offered  by  the 
innovation? 

Use  the  following  scale  for  estimating  importance: 

123456789  10 
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This  judgment  enables  the  various  benefits  that  may  be  derived  from  an 
innovation  to  be  weighted  against  each  other.  It  also  enables  a liability, 
if  identified  and  assessed  as  part  of  the  evaluation,  to  be  weighted 
against  other  liabilities,  and  against  benefits.  The  assessor  may  thus 
express  the  value  he  assigns  to  qualitative  influences  that  should  empha- 
size or  de-emphasize  quantitative  expressions  of  benefits  and  liabilities 
in  comparison  with  each  other,  with  a general  reference  to  the  Navy's 
training  and  operational  mission. 

Question  ID.  What  is  the  probability  of  achieving  the  estimated  magnitude 
of  the  benefit? 

In  Question  1A,  the  assessor  identified  the  magnitude  of  the  benefit  esti- 
mated to  occur  under  moderately  favorable  conditions.  The  assessor  now 
makes  the  following  two  estimates. 

a.  What  is  the  estimated  magnitude  of  the  benefit  under  conditions  that 
would  be  most  favorable?  This  value  should  exceed,  by  some  amount, 

the  estimate  of  the  magnitude  of  the  benefit  under  moderately  favorable 
condi tions. 

b.  What  is  the  estimate  of  the  magnitude  of  the  benefit  under  conditions 
that  would  be  highly  adverse  to  it?  This  value  should  be  less  than 
the  estimate  of  the  magnitude  of  the  benefit  under  moderately  favorable 
conditions.  The  value  could  become  a negative  one,  in  which  case  the 
proposed  innovation  would  be  a liability  with  regard  to  this  variable. 
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2.  Answer  questions  about  reductions  in  attrition  rate  during  and  at  the  end 
of  training. 

The  innovation  may  be  aimed  only  at  a reduction  in  training  time,  holding 
attrition  rate  constant.  Or  the  innovation  may  be  aimed  only  at  a reduc- 
tion in  attrition  rate,  while  holding  training  time  constant.  The  third 
alternative  may  attempt  to  combine  reductions  both  in  training  time  and 
attrition.  The  topic  here  applies  to  the  second  and  third  alternatives. 

Estimation  on  change  in  attrition  rate  should  focus  on  each  training 
element  or  unit  found  relevant  to  the  innovation.  The  standard  formula 
used  to  express  attrition  rate  in  Navy  training  will  define  the  present 
meaning  of  attrition.  One  should  also  assume  that  present  standards  or 
criteria  of  acceptable  performance  would  apply  to  the  innovation. 

Question  2A.  For  each  training  element  (or  unit)  applicable  to  the  innova- 
tion, what  is  the  estimated  percentage  reduction  in  existing  attrition  rate 
(or  normally  projected  attrition  rate)  to  be  expected  from  application  of 
the  innovation? 


Because  attrition  rates  are  expressed  in  percentage  figures,  the  estimate 
will  be  a percentage  of  a percentage.  Thus,  if  the  normal  attrition  rate 
for  course  X is  20%,  a reduction  of  the  attrition  rate  by  30%  would  reduce 
the  existing  rate  by  approximately  a third,  to  14%.  But  because  proportions 
seem  generally  easier  to  estimate  than  actual  values,  estimate  the  percent- 
age (or  proportion)  of  reduction  of  trainees  normally  attritloned  that  would 
not  be  attritioned  if  the  innovation  were  introduced.  Thus,  estimate  the 
percentage  reduction  in  the  existing  attrition  rates  if  this  Innovation 
were  to  be  applied  under  moderately  favorable  conditions. 

If,  however,  the  assessor  believes  that  under  conditions  moderately  favorable 
to  the  innovation,  the  attrition  rate  will  increase  rather  than  decrease  or 
remain  the  same,  he  should  go  to  question  2A  Negative. 

Question  2A  Negative 

Assume  that  the  assessor  believes  that  the  innovation  will  worsen  the  other- 
wise expected  attrition  rate.  He  would  specify  NEGATIVE  and  the  percentage 
of  attritioned  students  that  would  be  increased.  Thus,  the  expression 
NEGATIVE  25%  would  mean  that  a quarter  more  than  the  present  attritioned 
students  would  be  attritioned  if  the  innovation  were  introduced. 

He  would  answer  Questions  2B,  2C,  and  2D  as  they  are  put,  but  with  reference 
to  the  negative  value. 

Question  2B.  What  will  be  the  number  of  students  in  the  training  element 
or  unit  in  each  of  the  next  five  years? 
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Question  2C.  What  is  the  relative  importance  of  this  particular  benefit 
in  relation  to  all  of  the  other  benefits  (or  liabilities)  offered  by  the 
innovation? 


Use  the  following  scale  for  estimating  importance: 
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but  minor  substantial 


This  judgment  enables  the  various  benefits  that  may  be  derived  from  an 
innovation  to  be  weighted  against  each  other.  It  also  enables  a liability, 
if  identified  and  assessed  as  part  of  the  evaluation,  to  be  weighted 
against  other  liabilities,  and  against  benefits.  The  assessor  may  thus 
express  the  value  he  assigns  to  qualitative  influences  that  should  empha- 
size or  de-emphasize  quantitative  expressions  of  benefits  and  liabilities 
in  comparison  with  each  other,  with  a general  reference  to  the  Navy's 
training  and  operational  missions. 

Question  2D.  What  is  the  probability  of  achieving  the  estimated  benefit 
in  the  reduction  in  attrition? 


In  Question  2A,  the  assessor  identified  the  magnitude  of  the  benefit  esti- 
mated to  occur  under  moderately  favorable  conditions.  The  assessor  now 
makes  the  following  two  estimates. 

a.  What  is  the  estimated  magnitude  of  the  benefit  under  conditions  that 
would  be  most  favorable?  This  value  should  exceed,  by  some  amount, 

the  estimate  of  the  magnitude  of  the  benefit  under  moderately  favorable 
conditions. 

b.  What  is  the  estimate  of  the  magnitude  of  the  benefit  under  conditions 
that  would  be  highly  adverse  to  it?  This  value  should  be  less  than 
the  estimate  of  the  magnitude  of  the  benefit  under  moderately  favorable 
conditions.  The  value  could  even  shift  to  the  negative  side  and  be- 
come an  increase  in  attrition  rate  as  compared  with  present  attrition 
rate. 

3.  Answer  questions  about  reductions  in  aptitude  requirements. 

Assume  that  the  intent  of  the  innovation  is  to  reduce  the  normal  or  expected 
aptitude  requirements  for  achieving  course  objectives,  while  maintaining 
existing  training  time  and  attrition  rate  as  constants. 

In  this  context,  an  "aptitude"  is  defined  as  a score  on  standard  selection 
tests  that  are  used  for  the  classification  of  Navy  personnel,  which  is  a 
minimum  score  for  acceptance  to  a given  training  program.  By  lowering  a 
minimum  aptitude  level,  more  persons  become  potential  candidates  for  the 
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course  or  training  program.  If  the  cutoff  aptitude  scores  are  on  the  high 
slope  of  the  distribution  of  the  ability,  small  reductions  in  aptitude 
requirements  may  result  in  very  large  increases  in  potential  candidates. 

A reduction  in  aptitude  requirements  for  a given  training  objective  will 
be  valuable  if  the  Navy  sees:  (a)  shifts  in  overall  demand  for  higher 

level  aptitudes,  thus  increasing  competition  for  higher  aptitude  people; 
and/or  (b)  a shift  in  supply  of  recruits  towards  lower  aptitude  levels. 
Either  of  these  issues  presents  problems  that  are  not  subject  to  resolution 
solely  in  cost  justifications.  They  will,  therefore,  not  be  directly 
addressed  by  the  present  procedures,  although  the  issues  may  be  expressed 
qualitatively  in  the  following  format. 

It  is  possible  within  a given  training  context  to  assess  the  value  of  the 
innovation  applied  to  aptitude  level  in  terms  of  training  time  and  attri- 
tion rate.  Assume  that  the  innovation  claims  to  be  able  to  reduce  the 
relevant  aptitude  scores  by  "5  points".  In  existing  training  procedures 
(without  the  innovation),  the  people  within  these  5 points  on  the  apti- 
tude measures  would  require  an  average  of  20%  more  training  time,  as 
compared  with  the  average  training  time  of  those  students  within  present 
aptitude  levels.  In  addition,  the  new  group  would  have  an  attrition  rate 
of  30%,  whereas  the  attrition  rate  of  the  old  group  was  10%.  These  are 
cost  factors  which  can  be  used  to  evaluate  the  innovation. 

Assume  that  the  range-of-effect  analysis  has  identified  the  training 
courses  relevant  to  the  innovation.  It  is  also  assumed  that  the  minimum 
aptitude  scores  relevant  to  the  innovation  have  been  estimated.  If  this 
estimate  has  not  been  made,  this  estimate  should  be  a part  of  Question  3A. 

For  each  of  the  relevant  courses,  the  following  questions  should  be 
answered  in  making  both  quantitative  and  qualitative  assessments. 

Question  3A1.  What  is  the  proportion  of  students  in  the  innovated  class 
who  would  be  introduced  into  tne  class  if  the  innovation  were  to  be- 
appl ied? 

This  would  be  an  estimate  of  the  proportion  of  the  average  class  size  that 
would  fall  below  the  minimum  existing  aptitude  scores  and  above  the  new 
minimum  aptitude  score.  For  example,  the  range  of  the  reduced  aptitude 
levels  could  account  for  20%  of  all  the  students  in  the  revised  course 
makeup.  In  future  questions,  this  group  will  be  called  the  "affected 
students".  Assume  the  innovation  has  been  introduced  under  conditions 
moderately  favorable  to  it  when  making  this  estimate. 

Question  3A2.  What  would  be  the  average  additional  training  time  for  the 
affected  students  to  reach  the  criteria  of  the  course  objectives  on1 the- 
assumption  that  present  conditions  held,  and  the  innovation  were  NOT 
introduced? 
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It  is  possible  to  use  existing  experience  in  the  course  which  relates 
level  of  aptitude  to  class  performance  for  extrapolations  to  the  lower 
aptitude  levels. 

Question  3A3.  Assuming  the  additional  training  time  estimated  for  the 
affected  students  without  the  innovation,  what  would  be  the  estimated" 
attrition  rate  for  the  affected  students? 


It  is  recognized  that  it  is  very  difficult  to  make  judgments  about  two 
interacting  variables.  Nevertheless,  there  will  be  a proportion  of 
students  who  would  have  to  recycle,  even  with  an  extended  training  time 
for  the  average  of  the  affected  group.  Furthermore,  there  would  be 
inevitable  washouts  who  couldn't  learn  the  material  even  with  very 
extensive  training. 

Question  3B.  For  each  of  the  next  five  years,  what  is  the  number  of 
students  in  the  class  of  course  of  instruction  affected  by  the  innovation? 

Question  3C.  What  is  the  relative  importance  of  this  particular  benefit 
in  relation  to  all  of  the  other  benefits  (or  liabilities)  offered  by  the 
innovation? 


Use  the  following  scale  for  estimating  importance: 
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This  judgment  enables  the  various  benefits  that  may  be  derived  from  an  in- 
novation to  be  weighted  against  each  other.  It  also  enables  a liability, 
if  identified  and  assessed  as  part  of  the  evaluation,  to  be  weighted 
against  other  liabilities,  and  against  benefits.  The  assessor  may  thus 
express  the  value  he  assigns  to  qualitative  influences  that  should  empha- 
size or  de-emphasize  quantitative  expressions  of  benefits  and  liabilities 
in  comparison  with  each  other,  with  a general  reference  to  the  Navy's 
training  and  operational  mission. 

Question  3D.  What  is  the  probability  of  achieving  the  estimated  magnitude 
of  the  benefit? 


In  Question  3A,  the  assessor  identified  the  magnitude  of  the  benefit  esti- 
mated to  occur  under  moderately  favorable  conditions.  The  assessor  now 
makes  the  following  two  estimates. 

a.  What  is  the  estimated  magnitude  of  the  benefit  under  conditions  that 
would  be  most  favorable?  This  value  should  exceed,  by  some  amount. 
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the  estimate  of  the  magnitude  of  the  benefit  under  moderately 
favorable  conditions. 

b.  What  is  the  estimate  of  the  magnitude  of  the  benefit  under  conditions 
that  would  be  highly  adverse  to  it?  This  value  should  he  less  than 
the  estimate  of  the  magnitude  of  the  benefit  under  moderately  favor- 
able  conditions.  The  value  could  become  a negative  one,  in  which 
case  the  proposed  innovation  would  be  a liability  with  regard  to 
this  variable. 

4.  Answer  questions  about  supplementary  statements  of  benefits  and  liabilities 
This  would  be  Question  section  E to  be  added  to  Question  1 or  2 or  3, 
depending  on  which  was  answered  by  the  assessor  as  the  primary  target  of 
the  innovation.  This  section  is  also  a proper  supplement  to  benefit  evalua 
tions  derived  from  the  Instructional  Vehicle  assessment  route  (Subtasks 
05.02  to  05.08)  and  to  benefit  patterns  applied  to  on-the-job  benefits 
assessments. 

Study  of  the  innovation  in  the  context  of  the  evaluation  process  may  bring 
to  light  a number  of  liabilities  and  benefits  possibly  overlooked  in  the 
assessment  format.  The  assessor  is  invited  to  identify  such  additional 
benefits  and  liabilities,  state  them  qualitatively  and,  where  possible  or 
applicable,  quantitatively.  By  liability  is  meant  an  undesirable  outcome 
of  the  application  of  the  innovation  --  NOT  an  impediment  to  the  realiza- 
tion of  the  innovation.  The  assessor  may  see  long-range  effects,  or  side 
effects,  or  backlash  effects  not  directly  evident  in  the  innovation. 

The  assessor  should  also  review  the  data  obtained  in  Task  3,  Make  Prelimi- 
nary Feasibility  Profile,  and  reaffirm  or  modify  the  feasibility  factors 
as  evaluated  at  that  time.  These  factors  may  suggest  additional  benefits 
and  liabilities  inherent  in  the  innovation. 

As  in  his  other  estimates  and  predictions,  the  assessor  should  express,  as 
best  he  can  for  each  statement  of  benefit  or  liability  on  a variable,  an 
estimate  of  its  magnitude,  its  probability  of  happening,  and  its  relative 
importance.  An  outline  of  format  is  offered  in  Figure  5.18.1.  Where 
possible,  the  variable  or  factor  which  names  the  benefit  or  liability 
should  be  expressed  as  a qunatifiable  entity,  and  an  estimate  of  the 
magnitude  of  unit  benefit  or  unit  liability  be  expressed.  The  size  of 
the  multiplier  of  this  unit  benefit  should  be  determined  or  estimated  so 
that  the  full  scope  of  the  benefit  can  be  expressed  in  the  same  way  that 
the  preceding  variables  (course  length,  attrition  rate,  and  aptitude  level) 
have  been  expressed.  This  would  enable  formal  analysis  to  be  made. 

But  even  where  quantitative  magnitudes  and  the  numerical  size  of  the 
multiplier  factor  are  not  feasible,  the  identification  of  the  supplemental 
benefit  or  liability  should  be  made  and  accompanied  by  estimates  of  proba- 
bility of  occurrence  and  relative  importance.  Such  statements  may  not  be 
processed  formally  through  cost  models,  but  presented  to  the  decision  maker 
in  the  form  on  which  they  are  set  down  by  the  assessor. 
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Use  Figure  5.18.1  as  a structure  for  describing  the  supplemental  benefit 
or  liability.  It  is  probably  not  desirable  to  make  a printed  form  for 
this  information  as  the  assessor  may  be  constrained  from  getting  and  setting 
down  contextual  information  essential  to  understanding  the  message  because 
he  is  inhibited  by  lack  of  space  or  lack  of  suitable  category  heading. 

COMMENT:  The  data  developed  in  response  to  the  previous  procedural  questions 
should  be  prepared  and  formatted  according  to  the  demands  of  later  tasks  to 
which  they  are  input.  These  include  cost-benefits  analysis  and  economic 
analysis. 

5.  Transmit  data  developed  in  this  subyask  to  Task  6,  Perform  Cost-Benefits 
Analysis. 
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1.  Identify  the  name  of  the  variable  or  factor  which  is  a supplemental  bene- 
fit or  a liability. 

2.  Estimate  the  magnitude  of  effect  of  the  benefit/liability. 

a.  If  possible,  quantify  the  unit  of  magnitude,  and  the  multiplier 
factor,  to  apply  to  the  unit  of  magnitude  appropriate  to  the 
intended  scope  of  the  application  of  the  innovation. 

b.  If  numerical  quantification  is  impractical,  express  magnitudes 
in  relative  terms. 

3.  What  is  the  number  of  entities  or  instances  (jobs,  courses,  students,  etc.) 
to  which  this  magnitude  of  benefit  (or  liability)  applies? 

4.  Estimate  the  relative  importance  of  the  added  benefit  (or  liability)  to 
the  overall  effectiveness  of  the  innovation. 

Use  a scale  from  1 to  10. 
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Any  additional  statements  that  may  add  or  qualify  information  for  the 
decision  maker  may  be  added  informally. 

5.  Estimate  the  range  across  which  the  magnitude  of  benefit  (or  liability) 
may  occur. 

a.  The  estimate  of  magnitude  made  in  item  2 presumably  assumed 
conditions  moderately  favorable  to  the  benefit  occurring  (or 
moderately  unfavorable  for  the  liability  to  occur)  at  that 
magnitude. 

b.  Estimate  the  magnitude  of  the  benefit  under  conditions  highly 
favorable  for  the  innovation.  (The  converse  for  a liability.) 

c.  Estimate  the  magnitude  of  the  benefit  under  conditions  highly 
adverse  to  the  innovation. 

6.  State  reasons  and  justifications  for  these  expectations. 


FIGURE  5.18.1  FORMAT  FOR  DESCRIBING  A SUPPLEMENTARY 
BENEFIT  OR  LIABILITY 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  TYPE  OF  KNOWLEDGE 


DESCRIPTION 

A key  factor  in  the  economics  of  training  is  that,  unlike  most  skills,  most 
knowledge  can  be  learned  by  demanding  only  symbolic  responses  from  the  student. 
The  presentation  may  range  from  text,  diagrams,  photographs,  and  motion  pic- 
tures, to  computer  generated  displays.  But  the  students  write  or  vocalize 
answers  or  push  answer  buttons.  Knowledge  is  symbolic  information  about 
things  and  actions,  and  rules,  whereas  a skill  is  manifest  in  actions  per- 
formed as  tasks  with  purposes  and  goals.  Knowledge  may  support  skill,  but  is 
not  equivalent  to  skill. 

Unlike  skill  training,  which  is  more  or  less  directly  tested  by  its  applica- 
bility to  job  demands,  knowledges  can  be  tested  for  their  contribution  to 
learning  and  on-the-job  performance  only  by  elaborate  empirical  tests  which 
are  rarely  made,  and  usually  have  discouraging  results  unless  one  is  concerned 
with  where  to  cut  costs.  But  because  knowledge  training  is  relatively  inexpen- 
sive to  administer,  as  compared  with  skill  training,  it  widely  persists. 

Distinctions  should  be  made  between  reference  knowledge  and  enabling  knowledge. 

A reference  knowledge  is  information  context  that  is  not  directed  to  a task  or 
skill,  but  to  various  kinds  of  environments  and  ideas  in  which  the  worker  works 
and  the  job  is  performed.  Reference  knowledge  may  have  motivational  value,  such 
as  when  demonstrating  the  consequences  of  accidents  that  followed  violation  of 
safety  procedures. 

An  enabling  knowledge  is  one  which  directly  assists  in  the  learning  of  a task 
skill  or  supports  it  in  practice.  Thus,  the  ability  to  verbalize  a task  pro- 
cedure, while  not  the  task  skill  itself,  assists  in  learning  and  performing  it 
as  a skill.  Other  enabling  knowledges  consist  of  factual  data  that  are  used 
in  task  performance. 

A major  distinction  can  be  made  in  the  way  that  knowledge  content  is  learned, 
retained,  and  recalled.  A rote  knowledge  implies  that  verbal  lists,  tables, 
or  sentences  are  memorized  and  reinstated  In  a verbatim  sense  (multiplication 
table,  stanzas  of  a poem).  Materials  learned  by  rote  have  a restricted  range 
of  meaning,  so  that  the  content  of  rote  material  generalizes  very  little.  In 
general,  materials  of  this  kind  are  often  learned  by  repetitive  practice.  How- 
ever, they  may  be  better  retained  over  periods  of  time  when  supported  with 
mnemonic  images;  or  when  embedded  in  examples  and  other  "meaningful"  material. 

A second  form  of  knowledge  content  is  conceptual  --  it  deals  with  ideas  rather 
than  associations  between  symbols.  Concepts  are  abstractions  that  apply  to 
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classes  of  entities  and  their  properties.  The  meanings  of  terms  like 
"circuit",  "innovation",  and  "effectiveness"  are  the  nature  of  a concept. 
Concepts  are  subject  to  generalization  by  learning  and  by  thinking  pro- 
cesses. Concepts  can  be  organized  into  "principles"  and  higher  order 
concepts.  A learned  concept  has  a label  and  a meaning  or  reference, 
although  it  is  possible  to  exercise  the  "meaning"  of  a concept  without 
knowing  or  recalling  its  label. 

The  value  of  a knowledge  is  tested  not  by  its  mere  existence  in  memory,  but 
by  its  recall  as  a body  of  content  when  circumstances  would  make  its  recall 
useful.  Thus,  it  is  not  only  the  recall  of  the  knowledge  content  that  is 
of  practical  importance,  but  also  the  recall  of  the  applicability  of  the 
knowledge  to  a situation  other  than  to  a classroom  test  question.  As  an 
illustration,  the  knowledge  of  human  reaction  times  should  deter  the 
applied  psychologist  from  tailgating  drivers  ahead  of  him  at  high  speeds. 

It  is  possible  that  the  following  list  of  identifiers  may  assist  in  identify- 
ing the  courses  and  course  item  objectives  to  which  an  innovation  in  reference 
knowledge  training  might  apply.  Perhaps  a better  terminology  would  be  based 
on  traditional  names  assigned  to  these  kinds  of  training  objectives  by  Navy 
instructors  if  the  names  were  used  consistently.  A key  name  which  is  fairly 
general  is  "theory  of  operation"  which,  as  identified,  consumes  many  class- 
room hours  in  instructional  institutions  everywhere. 

Individual  facts  and  organized  clusters  of  knowledge,  meaningfully  connected, 
make  up  a body  of  reference  information  which  may  be  essential  to  the  perfor- 
mance of  skills  in  verbal  communication,  as  described  in  Subtask  05.13.  To 
this  learned  knowledge  should  be  added  skills  that  make  up  the  task  of 
"communicating  verbal  information".  As  indicated  in  the  following  subtask 
(05.20),  reference  knowledge,  often  of  more  limited  scope,  can  be  support- 
ive — if  not  always  essential  --  to  many  kinds  of  Navy  tasks,  including 
those  that  are  supervisory. 

The  reader  is  reminded  that  an  innovation  that  deals  with  cost  factors  asso- 
ciated with  a vehicle  of  instruction  --  such  as  devices  that  present  pictures 
of  things  in  motion  --  may  have  its  range-of-effect  more  efficiently  deter- 
mined by  the  Instructional  Vehicle  route,  Subtasks  05.02  through  05.07. 

PROCEDURE 

1.  Determine  whether  the  instructional  content  or  process  in  the  innovation 
is  applicable  to  a reference  knowledge  type  or  to  an  enabling  knowledge 
type.  Make  this  determination  on  the  basis  of  the  DESCRIPTION  under 
Subtask  05.20,  and  the  categories  described  in  Subtasks  05.20  and  05.21. 

2.  If  the  innovation  applies  exclusively  to  reference  knowledge  types,  pro- 
ceed to  Subtask  05.20. 
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3.  If  the  innovation  applies  exclusively  to  an  enabling  knowledge,  go  to 
Subtask  05.21 . 

4.  If  the  innovation  appears  to  apply  to  both  reference  and  enabling 
knowledges,  follow  the  route  beginning  with  Subtask  05.20  and  Subtask 
05.21. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  SELECT  APPLICABLE  TYPE  OF  REFERENCE  KNOWLEDGE 


DESCRIPTION 

A reference  knowledge  consists  of  information  about  the  contexts  of  the  per- 
sonal, technical,  and  organizational  situations  in  which  the  incumbent  and  his 
work  will  be  active.  Thus,  the  information  may  be  related  to  the  work  and 
work  environment,  and  perhaps  the  incumbent's  living  and  social  environment, 
but  does  not  constitute  a primary  work  skill,  nor  does  it  appear  to  have  any 
direct  bearing  on  the  learning  of  the  skill,  except  perhaps  by  motivational 
routes.  An  example  of  the  latter  might  be  satisfaction  in  being  identified 
with  important  and  prestigeful  organizations  and  functions,  or  the  opportunities 
for  promotions  and  career  development. 

Each  of  the  following  types  of  reference  knowledge  may  have  somewhat  different 
instructional  capabilities,  but  not  necessarily.  Obviously,  all  of  these  types 
could  be  transmitted  by  classroom  lecture  or  by  textbook.  But  imaginative 
techniques  could  be  devised  for  each  type  that  could  make  it  more  effective  — 
for  whatever  purposes  training  management  may  have  --  as  well  as  more  efficient 
than  the  traditional  alternatives. 

PROCEDURE 

1.  Determine  to  which  one  or  more  of  the  knowledge  types  listed  below  the 
innovation  is  applicable. 

2.  Even  if  the  innovation's  description  points  to  a given  type  of  course  con- 
tent, or  to  a given  named  course,  check  its  possibilities  against  the  other 
types  and  identify  them. 

• System  Purposes 

What  the  entire  organization  and  its  parts  are  supposed  to  do  and  the 
various  conditions  under  which  its  goals  are  to  be  accomplished.  The 
concept  of  mission,  and  mission  objectives,  and  of  the  values  to  be 
sought  and  maintained.  The  "system"  may  range  from  the  Department  of 
Defense,  the  Navy,  or  a branch  of  the  Navy,  to  the  ship  or  installa- 
tion and,  more  specifically,  to  the  purposes  and  objectives  of  the 
arm  and  watch.  The  form  of  instruction  may  range  from  platform  speeches 
to  motion  picture  scenarios  to  dynamic  simulations. 

• Organizational  Roles 

These  could  be  an  extension  of  system  purposes;  what  organizational 
entities  have  what  responsibilities  and  authorities  and  spheres  of 
influence.  Typically,  organizational  roles  are  communicated  with 
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organizational  charts.  Potentially,  far  more  meaningful  forms  of 
presentation  could  demonstrate  the  cooperative  and  competitive 
functions  of  hierarchic  and  scalar  relationships  in  organizational 
dynami cs . 

• Contexts  of  Operation 

The  interfaces  between  the  incumbent's  job  and  job  responsibilities 
that  impinge  on  his  own.  Also,  the  range  of  situations  and  circum- 
stances in  which  the  incumbent  is  expected  to  perform  his  tasks  in 
real  environments.  This  might  include  the  range  within  which  his 
initiatives  are  to  be  encouraged,  but  outside  of  which  constraints 
are  imposed  on  his  initiatives. 

• Theory  of  Operation 

This  may  range  from  theory  of  electronics  to  the  construction  of  and 
operating  functions  of  the  classes  of  equipment  the  maintenance 
mechanic  is  to  repair.  In  another  type  of  job,  dietary  theory  as 
potentially  applied  to  the  planning  and  preparation  of  meals. 

Theory  of  operation  generally  applies  to  how  something  works.  The 
content  may  be  developed  at  any  of  many  levels  of  discourse.  It  can 
serve  as  a basis  for  learning  of  nomenclature  and  identifications 
that  later  may  be  relevant  to  skill  learning. 

It  is  possible  that  the  learning  of  a suitable  kind  and  level  of  a 
theory  of  operation  may  enable  some  job  incumbents  to  deduce  task 
actions  in  the  absence  (or  forgetting)  of  the  applicable  procedure. 
This  assumption  may  underlie  instruction  in  theory  of  operation  to 
maintenance  mechanics  as  a substitute  for  troubleshooting  strategies 
and  procedures.  The  practical  justification  for  this  assumption  may 
be  debated  elsewhere. 

It  is  also  possible  for  the  learning  of  theory  of  operation  to  serve 
as  memory  aids  for  procedures,  both  in  their  learning  and  their  reten- 
tion and  long-term  recall.  In  order  for  this  assumption  to  be 
justified,  there  would  need  to  be  a careful  scheduling  of  instruc- 
tional sequences  such  that  the  two  kinds  of  learned  content  would 
effectively  support  each  other  in  the  student's  economics  of  learning. 

Since  theory  of  operation  generally  represents  some  kind  of  flow  or 
sequence  of  process  or  action,  a dynamic  display  is  suggested  where 
process  variables  may  be  changed  and  the  consequences  observed  in 
overall  system  behavior,  or  in  the  behavior  of  affected  components. 

To  the  extent  that  learning  theory  of  operation  does  facilitate  the 
learning  of  a skill  or  enhance  its  performance,  it  falls  into  the 
class  of  enabling  knowledge. 
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• Organizational  Rules  and  Constraints 

These  may  range  from  rules  governing  security  to  the  procedures  for 
ordering  a spare  part.  They  represent  realistic  factors  in  both  the 
working  environment  and  the  personal  environment  of  the  incumbent. 
Realistic  instruction  should  probably  include  the  kinds  of  conditions 
that  justify  the  breaking  of  a policy  and  the  means  of  making  the 
justification. 

• Nonwork  Directed  Content  of  Instruction 

A miscellaneous  category.  May  range  from  personal  hygiene  and 
recreational  facilities  to  interpersonal  psychology. 

3.  Specify  the  capability  of  the  innovation  to  support  one  or  more  of  these 
types  of  reference  knowledge,  and  the  key  function  or  capabilities  which 
enable  the  training  or  learning  enhancement  to  occur.  The  description 
should  be  structured  so  as  to  assist  in  identifying  courses  and  course 
content  in  which  the  innovation,  applied  to  these  or  other  reference 
knowledges,  may  be  applicable. 

4.  Examine  the  characteristics  of  enabling  knowledges,  Subtask  05.22,  bear- 
ing in  mind  the  possibility  of  applying  the  innovation  to  these  kinds  of 
instructional  content,  as  well  as  to  reference  knowledges. 

5.  Since  reference  knowledges  are  relatively  independent  of  specific  skill 
training  courses,  or  may  generalize  as  part  content  across  a wide  range 
of  skill  courses,  the  assessor  may  have  to  proceed  directly  to  the  data 
base  descriptions  of  all  training  courses  in  the  Navy  inventory.  "Theory 
of  Operation"  is  perhaps  the  only  term  which  is  fairly  standard  for  this 
content  in  all  courses  and  curricula,  at  least  in  equipment  oriented  train- 
ing. (But  even  courses  in  the  culinary  arts  have  a theory  of  dietary 
principles,  and  navigation  courses  often  contain  theory  of  celestial 
mechanics.)  This  state  of  affairs  places  a burden  on  the  assessor's  access 
to  knowledgeability  about  the  knowledge  content  of  courses  in  general,  and 
terminologies  used  for  course  objectives  that  apply  to  interpretations  of 
the  applicability  of  the  innovations. 

RATIONALE  AND  COMMENT 

The  issues  raised  in  step  5 make  up  the  general  difficulty  in  the  classifica- 
tion of  training  materials  that  do  not  quite  point  directly  to  skill  criteria, 
or  to  a pattern  of  steps,  each  one  of  which  has  a tangible  and  identifiable 
function  in  progressing  towards  a skill  objective.  The  characterization  of 
the  skill  objective,  being  subject  to  evaluation  in  an  operational  environment 
where  work  output  can  have  assignable  value,  simplifies  a naming  technique  that 
is  fairly  unambiguous.  By  contrast,  the  naming  of  knowledge  content  may  be 
somewhat  arbitrary  in  terms  of  its  inclusiveness  on  degree  of  detail. 
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The  assessor  may,  however,  find  another  dimensioning  of  knowledge  (cognitive) 
materials  useful.  They  are  related  to  R.  M.  Gagne's  taxonomy  of  building 
blocks  in  learning  (see  his  Conditions  of  Learning),  but  do  not  imply 
sequential  progression.  They  do  imply  differences  both  in  the  learning  process 
and  in  effective  instructional  technique.  In  brief,  the  classifications  are: 

Facts,  prescriptive  rules,  and  other  content  learned  by  repetitive  rote 
practice  or  "memorization".  Tables  of  information,  such  as  the  arithmetic 
tables,  or  one-for-one  codes.  The  rote  operations  may  be  embedded  in 
memory  aids  and  meaningful  contexts,  such  as  the  name  of  a device  with  a 
picture  of  description  of  its  use. 

Concepts,  which  are  ideas  that  have  generality,  such  as  of  the  properties 
of  things,  or  of  functional  relationships  among  things  or  processes,  or 
structures.  Thus,  "hierarchical  rank"  is  a concept,  whereas  "Always 
salute  a superior  officer",  is  a rule.  A concept  is  a basis  for  making 
equivalent  the  members  of  a class  of  things,  as  red  and  blue  are  equivalent 
under  the  concept  of  "color".  Concepts  have  to  do  with  the  meanings  of 
abstract  terms.  They  are  learned  by  abstracting  the  common  feature  among 
a collection  of  otherwise  different  experiences. 

Principles  are  combinations  of  concepts  in  the  form  of  sentences,  such  as: 

The  vehicle  moves  (a  concept)  when  its  wheels  have  sufficient  friction  or 
traction  on  the  roadway  (another  concept).  Principles  are  learned  and 
applied  by  linking  verbal  statements  to  a variety  of  experiences  which 
manifest  the  principle. 

Problem  solving  may  be  the  application  of  a structure  of  facts,  rules,  and 
principles  to  the  structure  of  a problem  situation. 

These  are  highly  condensed  characterizations,  each  of  which  justifies  a peda- 
gogical chapter  to  make  operational  for  the  purposes  of  instructional  analysis 
and  design.  It  is  possible  for  innovations  to  be  highly  applicable  (efficient 
and  effective)  for  one  of  these  classes  of  learning,  but  relatively  inapplic- 
able to  the  others.  Thus,  much  of  traditional  programmed  instruction  is  directed 
mainly  towards  the  learning  of  rote  content.  By  definition,  items  in  rote  con- 
tent have  little  context  (meaningful  associations)  attached  to  them. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 


SUBTASK:  IDENTIFY  TYPE  OF  ENABLING  KNOWLEDGE 


DESCRIPTION 

An  enabling  knowledge  is  one  which  facilitates  the  learning  of  a skill.  It 
may  or  may  not  be  relevant  to  the  exercise  of  the  skill  after  the  skill  has 
been  acquired.  Although  the  concept  of  enabling  knowledge  has  been  widely 
used  as  a rationale  by  educators  for  teaching  almost  anything  whatever  as  "pre- 
requisite" to  skill  training,  the  present  pragmatic  approach  to  the  acquisition 
of  on-the-job  skills  emphasizes  a direct  relationship  between  the  knowledge  and 
skill  acquisition.  The  following  categories  cover  the  major  range  of  enabling 
knowledges  that  are  quite  clearly  applicable  to  skill  learning. 

An  enabling  knowledge  tends  to  be  most  effective  when  it  is  acquired  imme- 
diately preceding  training  on  the  target  skill,  or  some  phase  of  it,  but  the 
economics  of  instruction  may  be  thought  to  justify  separation  in  time  and 
instructional  context  between  learning  the  knowledge  and  learning  the  skill  it 
should  support. 

PROCEDURE 

1.  Identify  the  type  of  enabling  knowledge  applicable  to  the  innovation. 

• Operational  Goal  Criteria 

The  variables  in  the  performance  of  skills  that  determine  goodness  of 
the  performance.  Some  of  these  criteria  may  be  immediate  and  local, 
others  may  be  remote  in  time  and  locale.  In  effect,  this  is  not  only 
what  good  versus  poor  performance  outcomes  look  like,  but  why  they  are 
important  to  the  incumbent,  to  his  organization,  and  to  the  Navy  mission. 
Realistic  instruction  may  include  the  specification  of  conditions  under 
which  some  criterion  performance  levels  may  be  relaxed  as  a trade-off 
against  no  performance  at  all.  If  such  material  is  shown  in  operational 
contexts,  perhaps  with  specific  examples,  the  student  may  acquire  sets 
of  values  that  enable  him  to  exercise  judgment  under  the  pressure  of 
conflicting  objectives  in  work  situations.  Verbal  abstractions  and 
exhortations  may  be  ineffective  in  transferring  to  the  work  situation. 

In  general,  learning  about  operational  goal  criteria  will  be  useful  to 
the  student  in  operations,  to  the  extent  that  he  can  relate  them  to 
choices  in  his  own  actions  at  the  time  he  must  select  the  action. 

• Nomenclature,  Identification,  and  Location  of  Work  Objects 

The  learning  of  the  names  of  things  associated  with  tasks,  what  they 
look  like,  and  where  they  are  located,  has  direct  transfer  value  to 
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the  learning  of  tasks  that  are  not  solely  manipulative.  The  better 
that  nomenclature  and  identifications  are  mastered,  the  more  readily 
can  the  student  apprehend  and  remember  instructional  content  that 
deals  with  that  nomenclature  and  associated  images  of  appearance  and 
location. 

Instruction  in  this  material  may  recognize  that  learning  is  faster 
and  retention  better  when  the  elements  are  learned  as  patterns  in 
contexts  that  are  meaningful  to  the  learner. 

• Procedure  Descriptions 

Procedures  may  be  described  at  many  levels  of  abstraction.  How 
efficient  the  learning  of  abstract  descriptions  of  procedures  may  be 
to  their  later  learning  at  the  level  of  detailed  steps  is  not  of 
concern  here.  Presumably,  any  image  that  the  student  acquires  about 
actual  operating  procedures  which  he  will  have  to  perform  may  have 
some  mnemonic  value  to  later  learning  of  the  actual  procedure  as  a 
skill.  When  the  description  of  the  procedure  to  be  learned  by  the 
student  is  highly  task  specific,  the  student  is  learning  what  is 
called  here  a "task  format",  which  is  treated  more  systematically  in 
the  section  of  that  name.  Subtask  05.22. 

Procedures  applicable  to  tasks  may  extend  to  action  policies,  principles, 
and  strategies.  Instruction  here  may  consist  of  what  these  policies  and 
principles  are,  rather  than  in  practicing  them  in  simulations  of  task 
performance.  Knowledge  about  these  policies  and  strategies  may  facilitate 
learning  them  in  practice  as  a skill. 

It  should  again  be  underscored  that  an  enabling  knowledge  --  as  con- 
trasted with  a task  format  --  about  procedures  consists  of  generalities 
about  procedures  or  protocols,  and  not  the  specific  learning  of  the 
procedure  as  a literal  action  specification.  Thus,  if  the  student 
merely  skims  a procedure  specification  in  a handbook  of  instruction,  it 
would  be  appropriate  to  call  the  result  of  this  skimming  an  enabling 
knowledge.  But  so  would  be  a general  outline  of  the  procedure  which 
had  to  be  learned,  as  such,  to  a high  criterion  of  recall.  The  result 
is  a general  impression  of  how  the  procedure  is  performed. 

• Job  Relevant  Facts 

These  are  facts  and  data  directly  applicable  to  job  tasks.  These  range 
from  the  number  of  degrees  in  a compass  to  the  temperature  at  which  a 
soldering  flux  bonds  a given  metal,  the  calories  in  a pound  of  potatoes, 
or  the  electrical  resistance  denoted  by  various  color  bands  on  a 
resistor.  Each  of  these  examples  can  readily  be  tied  in  with  a class 
of  job  task.  The  relevance  of  a fact  to  a task  can  readily  be  deter- 
mined by  asking  if  the  task  can  be  effectively  performed  in  the  absence 
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of  knowing,  or  accessing,  the  fact.  Unfortunately,  care  is  not  always 
given  in  the  design  of  training  content  to  differentiating  facts  essen- 
tial to  the  task  from  those  that  are  gratuitous.  (An  innovation  might 
well  consist  of  a technique  for  making  such  distinctions.) 

2.  Procedure  cannot  be  specific  in  the  identification  of  task-enabling  know- 
ledge content  in  courses  of  instruction.  This  content  is  often  mixed  with 
other  instructional  content.  However,  many  training  courses  will  explicitly 
identify  "nomenclature",  or  a synonym,  as  the  training  objective  of  some 
unit  of  instruction.  There  may  be  equivalent  designations  of  "General 
Procedure"  in  some  courses,  expecially  those  for  duties  at  the  officer  level 
But  it  may  be  difficult  to  parse  out  course  content  that  is  directed  specif i 
cally  towards  instruction  in  "Operational  Goal  Criteria."  The  assessor  may 
require  extensive  knowledge  about  courses,  or  interpret  the  descriptions  of 
their  objectives  and  content  in  order  to  identify  enabling  knowledges. 

For  these  reasons,  no  high  degree  of  precision  in  estimating  the  potential 
effects  of  innovations  dealing  specifically  with  enabling  knowledge  can  be 
expected.  The  alternatives  are  either  detailed  manual  examination  of 
course  content,  or  very  gross  estimates  of  percentages  of  given  course 
content,  in  terms  of  student  hours,  spent  on  enabling  knowledges. 

3.  The  output  of  the  analysis  performed  in  this  subtask  should  be  passed 
through  Subtask  05.22,  Identify  Type  of  Task  Format,  in  order  to  check  as 
to  whether  the  enabling  knowledge  supports  instruction  in  task  formats. 
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SUBTASK:  IDENTIFY  TYPE  OF  TASK  FORMAT 


DESCRIPTION 

A task  format  Is  a structure  of  how  the  task  is  done.  One  can  learn  the  task 
format  without  being  able  to  perform  the  task  in  real  life  circumstances.  But 
if  the  task  format  is  genuinely  applicable,  the  learning  of  the  format  will 
generally  facilitate  learning  the  task  as  a skill.  This  does  not  assert  that 
learning  the  format  at  the  same  time  that  one  practices  the  skill  will  not  be 
more  efficient  than  the  two-stage  process.  And  the  way  in  which  a format  is 
learned  may  be  significant  in  its  value  to  learning  the  skill. 

Task  formats  may  be  general  or  they  may  be  specific.  Thus,  the  idealized 
general  format  for  using  a test  instrument  consists  of:  (a)  determine  the 
variable  to  be  measured,  the  range  of  the  variable  to  be  measured,  and  the 
degree  of  precision  in  measurement;  (b)  locate  the  test  instrument  and  identify 
it  by  name,  and  check  its  function  and  capability  according  to  the  purposes  of 
the  moment;  (c)  check  its  calibration  ...»  and  so  on  through  its  use  and 
return.  These  are  conceptualized  set  of  steps  that  apply  to  the  use  of  any 
test  instrument.  It  is  an  example  of  a task  format. 

At  another  level,  a task  format  may  be  a verbalized  specification  of  every 
detailed  step  to  take  in  selecting  and  operating  a specific  voltmeter.  The 
verbalized  instructions  are  not  the  equivalent  to  the  ability  to  execute  the 
self-instructions  in  practice,  but  they  may  serve  as  a guiding  template  for 
carrying  out  the  action. 

A printed  requisition  form  embodies  the  task  format  of  preparing  a requisition 
for  one  or  more  objects  or  services.  The  names  of  the  blanks  to  be  filled  in 
are  the  variables  in  the  transaction.  The  entire  task  format  for  making  a 
requisition  may  include  finding  a copy  of  the  form,  filling  it  in,  and  trans- 
mitting the  order  form  to  the  purchase  agent. 

The  learning  of  a format  enables  a person  to  copy  with  a variety  of  different 
situations  that  fit  the  variables  that  make  up  the  format.  The  concept  of 
task  format  (whatever  the  concept  may  be  called)  is  important  to  the  economics 
and  practices  of  training  and  transfer  of  training.  It  is  not  a novel  notion 
in  educational  and  training  practice,  but  its  role  in  training  technology  should 
be  made  explicit.  Instruction  in  task  formats  is  relatively  inexpensive  in 
training  equipment  because  It  can  generally  be  accomplished  with  symbolic  exer- 
cises. The  sections  in  this  report  that  deal  with  short-term  memory  and  with 
decision-making  (Appendix  A on  Task  Structure  Elements)  amplify  the  meaning 
and  application  of  the  task  format  concept.  Appendix  A also  contains  an  analytic 
essay  on  task  formats. 
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A task  format  is  a skill  component,  therefore  it  is  set  apart  from  the  topic 

Enabling  Knowledge. 

PROCEDURE 

1.  Determine  whether  the  innovation  is  aimed  at  instruction  in  some  one  or 
another  type  of  task  format.  This  decision  will  be  based  on  the  general 
definitions  of  task  format  contained  in  this  subtask  under  DESCRIPTION, 
Appendix  A,  and  in  the  topics  under  step  2.  If  the  innovation  is 
inapplicable  to  task  formats  in  the  specific  sense  described  here,  go  on 
directly  to  Subtask  05.14. 

2.  Determine  the  type  or  types  of  task  format  to  which  the  innovation  applies. 
The  following  are  fairly  exhaustive  of  format  types  with  respect  to  cogni- 
tive tasks,  or  tasks  primarily  directed  by  cognitive  processes. 

• Procedure  Formats 

A task  format  may  serve  any  one  or  combination  of  the  following 
functions:  a structure  for  searching  for  task  information;  for 

apprehending  and  retaining  task  information  in  working  or  short- 
term memory;  for  accessing  the  relevant  content  in  long-term  memory; 
for  processing  task  information;  and  for  executing  the  work  output 
required  of  tne  task. 

By  this  definition,  task  formats  may  apply  to  any  one  or  combina- 
tion of  Task  Structure  Elements  (Subtask  05.13).  A given  work 
procedure  may  emphasize  some  one  of  these  task  elements,  but  many 
procedures  consist  of  a series  of  steps  in  sequence.  At  one  level 
of  format,  the  student  memorizes  the  verbal  statements  that  comprise 
the  steps  in  the  procedure.  Another  kind  of  format  — which  may 
support  the  first  --  is  his  memorization  of  a set  of  pictorial 
images  (such  as  from  a series  of  diagrams,  slides,  or  a motion 
picture)  of  the  sequence  of  steps.  Another  support  may  be 
demonstrations  of  the  procedure  by  the  instructor  with  actual  work 
objects.  To  the  extent  that  these  various  "encodings"  of  the  pro- 
cedure are  consistent  with  each  other,  they  strengthen  each  other 
in  the  student's  memory  of  the  procedure,  and  his  potential  ability 
to  translate  the  format  into  live  action.  But,  because  the  student 
may  not  have  an  action  repertory  to  match  the  symbolic  representations 
of  those  actions,  he  cannot  perform  the  actual  procedure  at  acceptable 
skill  levels  without  live  practice. 

Task  formats  may  be  learned  to  some  useful  degree,  independent  of 
practice  in  the  live  task  itself,  with  real  work  objects  in  real 
situations.  But  the  learning  of  task  formats  is  not  necessarily 
independent  of  learning  the  task  by  direct  practice  on  it.  Beyond 
given  levels  of  learning  on  a given  task  format,  it  is  likely  that 
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additional  practice  on  the  task  format  will  have  diminishing  returns 
in  transfer  and  savings  effects  on  the  learning  of  the  live  task. 

The  smaller  the  demand  by  the  task  on  complex  information  processing 
by  the  student,  the  smaller  the  relative  amount  of  practice  on  the 
task  format  that  will  be  useful.  Sequential  procedures  where  each 
step  is  a relatively  simple  situation-response  unit  demands,  by 
definition,  relatively  little  information  processing  except  that 
of  the  sequence  of  steps. 

In  conclusion,  many  procedure  formats  will,  in  practical  terms,  consist 
of  the  verbalization  and  imagery  of  procedural  steps  that  comprise  the 
task.  The  "rules  of  play"  for  the  game  in  chess  is  an  example  of  one 
level  of  task  format  for  the  game. 

• Decision  Formats 

Because  decisions  are  information  processing  operations,  decision 
formats  and  exercises  in  decision-making  can  be  accomplished  apart 
from  live  task  settings.  There  are  exceptions,  such  as  when  the 
decision  is  a progressive  one  and  depends  on  interaction  between  the 
incumbent's  overt  behavior  and  the  response  of  the  environment  to  his 
behavior.  But  even  in  such  cases,  limited  decision  formats  may  use- 
fully be  learned  independent  of  live  practice.  In  fact,  this  is  what 
many  Instructors  attempt  to  do  when  briefing  a student  in  advance  of 
a maneuver  in  what  he  should  attend  to  and  be  prepared  to  do.  The 
briefing  establishes  at  least  a temporary  format  for  the  performance. 

A decision  format  is  primarily  a transactional  template  of  (a)  the 
variables  that  make  up  a given  type  of  problem  situation,  (b)  the 
variables  in  the  relevant  response  set  from  which  an  alternative  is 
to  be  chosen,  and  (c)  the  rules,  policies,  or  strategies  for  select- 
ing the  response  to  a given  problem  context.  A decision  format  may 
include  the  process  of  handling  absent,  ambiguous,  or  garbled  data 
about  one  or  more  problem  variables,  such  as  by  implicit  assignment 
of  probabilities  that  the  variable  has  some  hypothetical  value  in  the 
given  case.  The  once  popular  business-gaming  exercises  attempted  to 
teach  students  of  business  the  formats  of  marketing  decisions. 

Strategy  principles  for  selecting  a move,  or  a series  of  moves,  in 
chess,  are  an  example  of  relatively  complex  decision  formats  above 
the  level  of  rules  for  moving  chess  pieces. 

• Construction  Formats 

Whereas  decision-making  tends  to  be  a process  of  converging  on  the 
selection  of  a response  from  a limited  response  set,  constructing 
tends  to  be  both  a divergent  as  well  as  convergent  process.  The 
design  of  a pattern,  such  as  a plan  for  building  a bridge  or  for  an 
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attack  mission,  often  requires  the  construction  of  alternatives  as 
well  as  their  selection. 

3.  Specify  the  type  of  format,  and  make  a judgment  as  to  the  range  of  types 

of  job  tasks  for  which  the  innovation  might  be  applicable.  Note  that  all 
requirements  for  verbalized  learning  of  procedures  in  order  to  fulfill 
verbal  tests  of  procedural  competence,  point  to  what  is  here  called  a 
"procedure  format."  Any  training  objective  which  reads:  "Must  give  a 

detailed  verbal  description  of  the  procedure,"  identifies  the  learning 
of  a format  in  the  sense  applicable  here.  Training  in  decision  and  con- 
struction formats  is  less  well  understood  and  practiced. 

Key  terms  applied  to  job  tasks,  and  training  courses  that  may  be  appli- 
cable to  the  innovation  applied  to  decision-making  and  constructions 
include;  troubleshooting,  diagnosis,  fault  localizing,  planning,  schedul- 
ing, laying  out,  constructing,  problem  solving,  calculating,  and  plotting. 

4.  Pass  the  descriptions  generated  in  this  subtask,  as  well  as  information 
coming  from  Subtask  05.21,  Identify  Type  of  Enabling  Knowledge,  into 
Subtask  05.14,  Identify  Relevant  Jobs  and  Tasks. 

5.  The  alternative  can  be  to  use  the  descriptions  of  the  task  formats, 
applicable  to  the  innovation,  for  going  directly  to  the  data  base  of 
course  description,  bypassing  the  job  task  analytic  route. 
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SUBTASK:  ASSESS  ON-THE-JOB  BENEFITS 


DESCRIPTION 

The  making  of  rigorous  cost-benefit  assessment  of  improvements  in  on-the-job 
performances  of  various  kinds  follows  the  same  principles  that  were  used  for 
developing  and  applying  to  functions  in  the  training  enterprise.  But  the  gap 
between  principle  and  practice  has  not  been  bridged  for  formal  on-the-job 
analysis. 

However,  lack  of  formalized  analytic  techniques  should  not  deter  useful  judg- 
ments to  be  made  in  practical  circumstances  where  something  needs  to  be  done, 
and  an  informed  guess  is  better  than  nothing  at  all. 

The  innovation  may  not  be  justified  by  benefits  achieved  solely  within  the 
training  organization,  such  as  by  reduction  in  displaceable  costs  in  instruc- 
tional vehicles,  reduction  in  training  time,  or  reduction  in  aptitude  levels. 

In  consequence,  the  justification  of  the  innovation  will  depend  on  its  making 
a worthwhile  contribution  to  graduates  doing  their  jobs  in  actual  job 
environments . 

A rigorous  cost  justification  of  increased  job  competences  derived  from  analysis 
in  the  operational  environment  is  rarely  feasible,  and  when  it  is,  critical 
variables  will  be  omitted.  Examples  of  such  variables  may  be  morale,  depend- 
ability, readiness,  initiative,  dedication,  and  similar  abstractions  which 
although  having  subjective  meaning  and  sometimes  even  predictive  validity  when 
based  on  subjective  evaluations,  fail  in  objectivity  necessary  for  cost  and 
value  analysis. 

The  following  is  a nonexhausti ve  list  of  types  of  on-the-job  benefits  which  may 
result  from  innovations  that  "improve"  the  product  of  training:  on-the-job 
skills  and  competences.  It  is  assumed  that  the  innovation  has  been  put  through 
the  range-of-effect  analysis,  and  that  tasks  and  jobs  relevant  to  the  innovation 
have  been  identified  quite  specifically.  From  this  identification,  the  number 
of  incumbents  to  which  the  innovation  is  now,  or  potentially,  applicable  can  be 
quantified.  Having  identified  the  one  or  more  types  of  benefit  applicable  to 
the  result  of  the  innovation  in  a job  skill,  the  assessor  follows  the  same  format 
for  describing  benefits  as  outlined  in  05.18. 

In  this  case,  the  assessor  is  not  likely  to  be  a training  officer,  but  an  opera- 
tions officer  familiar  with  the  relevant  job  skills  and  the  work  environments  in 
which  they  are  performed.  Only  such  individuals  are  able  to  make  useful  judg- 
ments of  magnitudes  and  importance  of  given  enhancements  to  specific  skills, 
and  the  probability  of  realizing  the  benefit  under  actual  work  conditions.  Thus, 
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if  a person's  duties,  as  they  exist,  tend  to  demand  his  full  time  and  attention, 
there  is  little  benefit  to  be  gained  from  giving  him  skills  which  demand 
additional  time,  or  that  require  him  to  be  elsewhere.  Nor  is  there  point  to 
providing  him  with  a skill  to  be  exercised  where  the  tools  necessary  for  exer- 
cising the  skill  are  unavailable.  A potential  benefit  must  have  an  opportunity 
to  happen  in  order  for  it  to  become  an  actual  benefit. 

Having  completed  the  analysis  of  a potential  benefit  derived  from  an  innovation, 
an  attempt  may  be  made  to  determine  its  value  in  various  kinds  of  savings  in 
operational  environments,  such  that  these  savings  can  be  translated  into  dollars. 
Examples  of  such  savings  may  be  in  terms  of: 

Reduction  of  manpower  units  in  the  operating  organization. 

Reduction  in  tools,  supplies. 

Reduction  in  expendable  materials  wasted  per  unit  of  mission 
effectiveness;  e.g.,  wasted  rounds  per  kill. 

Reduction  in  units  of  standby  equipments  for  operational  readiness. 

Reduction  in  probability  of  catastrophic  failures;  e.g.,  personnel 
casualties,  planes  lost,  ship's  engines  crippled,  collision  at  sea, 
major  oil  spill,  nuclear  diffusion  to  open  environment. 

But  where  savings  of  this  kind  cannot  be  determined  or  are  inadeauate  to  account 
for  the  benefit,  a subjective  evaluation  of  the  benefit  must  be  made  by  responsi- 
ble officers.  This  evaluation  of  the  benefit  must  then  be  weighed,  subjectively, 
against  the  costs  of  the  innovation  and  the  probabilities  of  its  being  realized. 

If  the  added  cost  to  training  incurred  by  the  innovation  must  work  within  fixed 
training  and  other  personnel  resources,  then  the  evaluating  officer  may  have  to 
make  a second  order  decision:  what  to  sacrifice  in  the  existing  products  of 

training  to  compensate  for  the  added  cost  of  the  innovation.  This  kind  of 
decision  is  typical  of  management  that  operates  under  the  constraints  of  budgets. 

The  following  are  types  of  benefits  from  enhanced  skills. 

Greater  flexibility  in  assignment. 

Less  on-the-job  training  required  for  acceptable  work  performance. 

Higher  productivity  levels,  higher  quality  of  performance  (fewer  errors). 

Greater  range  in  work  competence  --  in  situations,  environments. 

Willingness  under  stress  — increased  motivations  to  continue  working 
under  adverse  conditions  of  discomfort,  hazard,  fatigue. 
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Initiative:  increased  capability  for  self-initiated  and  self-directed 

activities. 

The  use  of  one  or  more  of  these  descriptive  titles  should  be  augmented  with 
specifics  related  to  tasks  and  task  contexts  in  the  description  of  the  benefit. 

PROCEDURE 

1.  The  assessment  of  the  benefit  should  be  completed  by  using  the  "Format  for 
Describing  a Supplementary  Benefit  or  Liability"  in  Subtask  05.15. 
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TASK:  DETERMINE  RANGE-OF-EFFECT 

SUBTASK:  IDENTIFY  RELEVANT  FUNCTION  IN  DEVELOPMENT  AND  ADMINISTRATIVE 

MANAGEMENT  OF  TRAINING 


DESCRIPTION 

The  giving  of  instruction  is  preceded  and  surrounded  by  many  kinds  of  support 
activities.  Although  some  of  these  activities  may  be  peripheral  to  training 
functions,  many  of  them  are  mainstream  and  essential.  These  activities  can  be 
clustered  into  nodes  or  functional  tasks  in  the  support  of  training.  As  pro- 
cessing functions,  they  receive,  process,  and  output  decisions  and  other  classes 
of  work  information. 

Much  of  the  information  developed  as  output  from  one  node  of  task  activity  in 
the  training  enterprise,  is  passed  along  as  input  to  another  node.  Thus,  task 
analysis  and  job  requirements  and  specifications,  as  an  output,  become  partial 
input  to  another  node  which  develops  "course  specifications"  as  an  output. 

This,  in  turn,  passes  into  a node  which  generates  "course  content  analysis  and 
development."  Finally,  this  sequential  network,  with  contributory  branches, 
results  in  instruction  to  students  which,  hopefully,  results  in  learned  skills, 
which  in  turn  result  in  on-the-job  performance. 

With  some  exceptions,  this  network  of  nodes  seems  rather  loosely  coupled.  That 
is,  kinds  and  categories  of  Information  that  function  mode  A should  produce  as 
input  to  node  B in  order  to  precisely  meet  the  requirements  of  function  B,  is 
generally  vague.  Thus,  much  of  what  B ought  to  have,  it  does  not  get  from  A. 

And  much  of  what  it  does  get  from  A,  it  doesn't  need  or  know  how  to  use.  B's 
difficulties  stem  partly  from  not  being  tightly  coupled  to  function  node  C.  B 
does  not  know  precisely  what  input  C needs  to  do  its  job,  and  this  uncertainty 
is  transferred  back  to  the  relationship  of  B to  A. 

This  loose  coupling  of  information  structures  and  relationships  presents  great 
problems  in  any  attempt  to  evaluate  an  innovation  which  claims  to  "improve"  the 
effectiveness  of  node  function  B.  If  the  innovation  merely  claims  to  do  what 
B does  now,  and  no  better  or  no  worse  in  terms  of  input  demand  or  output  product, 
then  the  innovation  can  be  evaluated  only  on  whether  it  saves  costs  in  perform- 
ing or  supporting  function  B.  If  reducing  the  cost  of  the  B operation  imposes 
a greater  cost  on  what  A must  produce,  however,  then  the  savings  must  be  the 
net  from  the  savings  effected  in  B,  minus  the  increased  costs  in  A. 

But  suppose  that  the  innovation  claims  to  improve  the  "quality"  of  B's  output 
to  function  C,  but  at  an  increase  in  the  cost  of  B.  The  benefit  must  be 
assessed  by  what  happens  in  C.  If  C's  output  to  D is  not  improved  as  a direct 
result  of  B's  improvement,  the  claim  of  improvement  in  B cannot  reasonably  be 


III-156 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
05  24  02 


substantiated,  with  this  exception:  the  cost  of  function  C is  reduced  as  a 

consequence  of  the  new  kind  of  output  from  B.  This  chain  of  referral  of 
"benefit"  ultimately  ends  in  the  performance  of  the  student  after  he  gets  on 
the  job,  and  in  assessing  improvements  in  that  performance  that  can  feed  back 
to  justify  the  increase  in  the  cost  of  function  B.  The  more  tightly  coupled 
the  functional  nodes  are  to  each  other,  the  more  readily  can  such  evaluation 
be  made. 

This  observation  should  make  it  clear  why  many  innovations  that  have  high  face 
validity  for  improving  a function  in  the  training  subsystem,  are  virtually 
doomed  unless  there  are  corresponding  changes  all  along  the  line,  or  unless 
the  innovation  really  eliminates  duplicated  work  among  various  nodes  --  but 
this  should  really  be  evaluated  as  a cost  saving,  an  improvement  in  the 
efficiency  of  what  is  done  now,  and  not  an  improvement  in  the  quality  of  the 
"service.7'  It  should  be  noted  that  in  actual  training  operations  there  is  a 
common  reference  criterion  of  "quality":  how  much  did  the  student  learn  and 

perform  per  unit  of  cost?  This  ultimate  criterion  is  measurable,  hence  subject 
to  assessment  against  a cost  base. 

Intuitively  at  least,  there  seems  to  be  massive  potential  for  improving  the 
quality  and  cost  picture  in  the  many  supporting  activities  that  are  here  called 
"Development  and  Administrative  Management  of  Training"  (D&A).  But  without  the 
ability  to  trace  through  the  improvement  in  quality  of  output  in  a given  D&A 
function  to  a knowledge  or  skill  product,  the  assessment  of  changes  having  the 
intent  to  reflect  in  improved  quality  of  operations  must  be  limited  to  demon- 
strations of  cost  improvements,  or  to  quite  intuitive  hunches  that  somehow 
better  training  will  be  the  result.  But,  because  the  system  can  readily  absorb 
local  changes  without  overt  effects,  the  hunches  are  likely  to  be  overestimates. 

A process  model,  therefore,  needs  to  be  developed  which,  at  some  practical 
level  of  precision,  can  enable  an  innovation  in  technique  or  tool  made  in 
function  B not  only  to  be  measured  in  costs  to  B and  all  of  the  functions  that 
feed  or  are  fed  by  B,  but  also  to  relate  the  change  in  B to  an  ultimate  quality 
benefit  in  student  skills.  It  is  naive  to  assume  that  such  a process  model  can 
be  as  rigorous  as  a computer  algorithm,  but  it  need  not  be.  Part  of  the  art  in 
developing  such  a model  would  be  knowing  when  to  stop  in  seeking  precision,  and 
when  a practical  level  of  precision  had  been  reached. 

These  considerations,  therefore,  lead  to  focus  on  reducing  costs  in  given  opera- 
tions within  a D&A  node,  or  among  a group  of  nodes.  Such  analysis  cannot 
differentiate  whether  the  reduction  in  cost  is  in  something  which  affects  a 
training  product  or  is  irrelevant  to  a training  product.  In  the  case  of  an 
exception,  where  the  assessor  can  relate  an  innovation  in  a function  directly 
to  a learned  product  In  on-the-job  performance,  his  assessment  of  the  latter 
will  enter  Subtask  05.10,  Identify  the  Learning  Objective,  and  proceed  through 
Subtasks  05.11,  05.12,  and  so  on,  through  05.18,  in  the  task  content  training 
path. 
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The  following  is  a list  of  D&A  functions  and  brief  characterizations  of  them. 

The  list  is  not  exhaustive,  and  the  innovation  may  not  fit  any  particular  one 
of  them.  This  is  not  particularly  important,  because  in  any  event  the  assessor 
would  have  to  localize  the  processes  or  products  in  the  D&A  network  which  the 
innovation  was  specifically  intended  to  replace  or  displace. 

• Capture  and  Description  of  Skill  and  Knowledge  Requirements 

This  is  a set  of  techniques.  Their  application  is  of  variable  quality 
and  utility,  although  invariably  time-consuming.  The  data  feed  into 
virtually  all  training  operations  that  follow  and  extend  into  job 
incumbent  assessment,  at  which  point  they  can  become  feedback  as 
"requirements"  data  to  training. 

• Matching  the  Skill  Requirement  With  Descriptions  of  Available  Resources 

This  is  a decision-making  situation.  The  operation  may  consist  of 
seeking  for  and  matching  a pattern  of  skills  requirement  with  existing 
skill  and  knowledge  in  trained  and/or  experienced  persons;  or  it  may 
be  an  attempt  to  match  the  requirements  with  a pattern  of  existing 
resources  in  the  form  of  aptitudes  to  learn  the  required  skills. 

• Matching  the  Skill  Requirements  With  Available  Training  Resources 

This  is  a decision-making  situation.  Training  resources  consist  of 
space,  facilities  to  house  and  teach,  existing  courses,  and  personnel. 

• Developing  Training  Requirements  From  Skill  Requirements  Descriptions 

A set  of  techniques.  The  training  requirements  picture  will  be  a 
breakdown  of  the  overall  skill  requirements  into  enabling  and  terminal 
instructional  objectives,  derivations  of  knowledges  to  support  skills 
or  skill  acquisition,  functional  requirements  of  instructional  vehicles, 
and  allowable  training  time  per  student. 

t Selecting  the  Training  Environments 

This  decision  may  precede  or  follow  the  specification  of  the  training 
requirements.  It  will  result  in  a choice  of  what  training  objectives 
are  to  be  acquired  in  formal  training,  and  which  ones  in  on-the-job 
training  and  experience.  It  may  also  select  the  school  where  the 
option  is  available. 

• Slow  Versus  Fast-paced  Development  of  Training  Material 

This  is  a decision  which  is  followed  by  technique.  On  occasion,  the 
demand  may  arise  for  getting  people  trained  at  a very  early  date,  so 
that  the  normal  processes  in  the  development  of  training  must  be  by- 
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passed  or  accelerated.  This  bypassing  may  be  costly  in  hard-to-find 
talent  and  other  training  resources. 

• Selection  or  Devising  of  Training  Modules 

A decision  followed  by  techniques  to  implement.  A training  module  may 
be  a course  used  as  an  element  in  a pattern  of  other  courses.  The 
module  thus  serves  more  than  one  purpose.  The  module  may  be  instruc- 
tional entities  within  courses  or  even  independent  of  courses.  A good 
modularity  is  one  which  enables  the  smallest  set  of  modules  to  be 
organized  to  ^erve  a wide  variety  of  purposes  with  a minimum  of  waste. 

• Selection  of  Instructional  Vehicles  and  Media 

A decision-making  operation.  The  result  is  not  merely  the  choice  of 
individual  vehicles,  but  a set  of  them  which,  together,  are  cost- 
effective  in  producing  a pattern  of  interlocking  skills  and  knowledge 
making  up  a duty,  job,  or  on-the-job  competence. 

• Designing  the  Content  of  Instruction 

Techniques  for  developing  the  content  of  what  the  instructional  vehicles 

deliver  to  the  student.  This  results  in  what  the  student  actually  sees, 

tries  to  learn,  and  does.  The  technique  may  be  general  to  any  training, 
or  restricted  to  a type  of  skill  or  knowledge  objective.  If  the  latter, 
further  analysis  of  range-of-effect  should  be  made  by  following  the 
procedures  in  Subtasks  05.19,  05.20,  and  05.21. 

• Devising  Selection  Procedures  for  Students 

Techniques  for  relating  the  level  and  range  of  training  objectives  for  a 

job  or  duty,  and  time  allowed  for  the  training  to  these  objectives  with 

entrant  capabilities  in  the  form  of  existing  levels  of  skill  and 
knowledge  acquired  by  training  or  on  the  job,  or  in  the  form  of  measured 
aptitudes  predictive  of  learning  rates  and  potential  terminal  skill 
levels.  Note  that  this  item  relates  to  the  previous  topic  on  "Matching 
the  skill  requirement  with  descriptions  of  available  skill  resources." 

• Assessment  of  the  Student 

Techniques  for  efficiently  sampling  from  student  skills  and  knowledge  at 
the  end  of  training,  with  respect  to  training  objectives  and  job 
requirements.  Also,  decisions  as  to  cutoff  levels  for  washouts. 

• Assignment  of  Graduates  to  Job  Positions 

Techniques  and  policies  for  matching  the  competences  (or  the  description 
of  competences)  of  graduating  students  with  the  identification  or 
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description  of  job  and  work  environment  assignments. 

• Evaluation  on  the  Job 

Techniques  for  efficiently  and  effectively  assessing  the  on-the-job 
competences  of  the  graduate,  the  extent  to  which  his  range  and  level 
of  performance  is  unacceptable,  acceptable,  or  superior,  and  the 
transcription  of  these  data  on  the  student's  record  for  use  in  his 
later  training  or  assignment,  and  for  use  in  the  revision  of  training. 

• Career  Development 

Techniques  for  the  organization  of  a progression  of  training  and  job 
experiences  that  are  psychologically  cumulative  and  operationally 
practicable,  and  decision  structures  for  the  matching  of  a career 
pattern  with  individuals  at  various  stages  of  training  and  experience. 

• Administrative  Planning  and  Control 

Decisions  and  techniques  for  the  planning  and  control  of  the  overall 
operations  of  the  training  enterprise,  or  substantial  segments  of  it. 
The  adapting  and  regulation  of  input  requirements  and  objectives  with 
existing  and  planned  resources,  according  to  organizational  policies 
and  values.  The  information  base  may  consist  not  only  of  current 
status  and  projections  of  immediate  future  status,  but  archival  data 
that  enable  experience  to  project  the  implications  of  trends,  and 
the  range  of  variability  in  system  behavior  within  which  the  perturba- 
tions will  damp  themselves  out,  but  outside  of  which  the  perturbation 
will  amplify  itself.  The  information  base  may  also  contain  long-term 
future  predictions  of  needs  that  enable  the  application  of  structural 
models  for  developing  game-playing  solutions. 

• Evaluation  of  Learning  or  Entropy  of  the  Training  System 

Techniques  for  assessing  and  evaluating  whether  the  rate  of  adaptive 
change  by  the  training  enterprise  as  an  operational  structure  is 
progressively  losing  control  of  its  overhead  costs,  the  relation  of 
process  to  product,  and  to  the  required  quality  of  the  product,  or 
whether  the  rate  of  adaptation  is  outpacing  the  entropy  characteristic 
of  very  large  organizations  over  a period  of  time.  This  function  is 
somewhat  outside  that  of  operational  administration. 

PROCEDURE 

1.  Using  the  foregoing  categories  as  a guide,  but  without  being  limited  to  the 
subject  matter  identified  by  them  for  D&A  functions,  identify  the  tasks  and 
task  activities  to  which  the  innovation  is  relevant,  both  as  a benefit  and 
as  a liability.  The  relevant  tasks  should  be  identified  as  precisely  as  is 
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practical  for  analytic  assessment  of  benefits  and  costs  to  be  derived 
from  the  innovation. 

COMMENT 

A useful  concept  for  characterizing  and  bounding  a task  consists  in  making 
a "transactional  structure"  of  the  task.  This  can  be  done  at  any  of  many 
levels  of  detail  or  generality.  The  structure  consists  of  the  following: 

a.  Source  of  input  information  to  the  task-function. 

b.  Variables  in  the  essential  input  "message  format"  that  feed  into  the 
function.  One  or  more  input  message  contexts  become  a "demand"  for 
service  from  the  task-function. 

c.  Variables  in  the  essential  output  information  or  message  format  from 
the  task-function  — the  output  represents  the  "service"  performed  by 
the  task. 

d.  Variables  in  the  reference  information  that  serve  as  supplemental 
input  to  the  task-function. 

e.  The  operations,  principles,  procedures,  rules,  and  problem-solving 
techniques  that  combine  input  message  contexts  and  reference  informa- 
tion, and  transform  them  into  output  messages  which  represent  the 
service  performed  by  the  function. 

Notice  that  the  basic  characterizations  of  input,  output,  and  reference 
information  are  in  the  variables  or  categories  of  information  that  comprise 
a message  context,  whether  of  an  input  demand  for  service,  or  reference 
data  for  generating  into  output,  or  in  the  output  Information  that  is  used 
as  a service.  The  categorical  structure  of  the  inputs  to  and  outputs  from 
a task  enable  the  tracing  of  functional  continuities  and  relevances  from 
one  task  to  another  in  a sequential  network.  This  coupling  enables  the 
effects  of  changes  in  one  part  of  the  network  to  be  predicted  (hence, 
assessed)  in  other  parts  of  the  network. 

2.  Pass  the  information  developed  in  step  1 on  D&A  tasks  relevant  to  the 
innovation  to  Subtask  05.18,  Assess  the  Training  Benefit  Pattern. 

COMMENT 

The  foregoing  descriptions  of  nodes  and  task-functions  of  D&A  activities 
are  essentially  the  descriptions  of  human  tasks,  since  they  are  performed 
by  job  incumbents,  and  supervised  and  monitored  by  humans  with  task 
responsibilities.  Procedural  step  1 will  have  identified  the  job  tasks 
in  D&A  operations  to  which  the  innovation  applies. 
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The  structure  for  assessing  the  benefits  of  on-the-job  tasks  is  presented 
in  Subtask  05.23,  in  the  section  dealing  with  benefits  that  an  innovation, 
which  leads  to  improvement  in  a skill  or  competence  can  have  for  the  job 
environment.  Techniques  for  the  quantitative  analysis  and  evaluation  of 
"improvements"  in  on-the-job  functions  are  as  applicable  to  D&A  jobs  as 
they  are  to  the  jobs  dealing  with  the  operation  and  maintenance  of  equip- 
ment, or  in  providing  clerical  or  professional  services. 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 

Task  3 (Make  Preliminary  Feasibility  Profile)  and  Task  4 (Perform  Analytic 
Feasibility  Assessment)  were  concerned  with  preliminary  assessment  exercises 
on  limited  range-of-appli cation  data.  Based  upon  a tentative  decision  to 
accept  the  proposed  innovation.  Task  5 (Determine  Range-of-Effect)  will  have 
identified  the  full  range  of  applications  and  will  have,  also,  provided  the 
full  pattern  of  benefits  (and  liabilities)  intrinsic  to  the  innovation.  The 
purpose  of  this  task  is  to  incorporate  these  two  additional  sets  of  informa- 
tion within  the  decision  process.  The  decision  tree  used  in  Task  4 will  be 
revised  to  include: 

• More  precise  cost/savings  data  for  each  outcome 

• Benefits/liabilities  data  stated  in  equivalent  dollars 
t Refined  success  probability  estimates 

• A sensitivity  analysis  identifying  critical  variables. 

Cost  data  derived  within  this  task  will  be  used  in  Task  7 (Perform  Financial 
Analysis).  It  should  be  pointed  out  that  two  separate  decision  patterns 
are  being  developed  to  assist  in  the  ultimate  decision-making  step.  The 
first  uses  the  decision  tree  where  an  attempt  is  made  to  quantify  benefits 
(including  dollar  savings)  and  liabilities  (including  dollar  costs)  and  then 
to  develop  a value  for  each  of  the  possible  decision  paths.  This  approach 
will  aid  the  overall  decision  process,  especially  when  normally  nonquantified 
benefits  and  liabilities  will  have  a substantial  influence  on  the  decision 
maker's  view  of  the  overall  acceptability  of  the  proposed  innovation.  The 
second  decision  pattern  (outlined  in  Task  7)  is  concerned  with  a straight- 
forward financial  analysis  of  the  proposal  using  identifiable  costs  and  savings 
over  an  extended  time  horizon.  There  the  decision  criteria  will  be  such 
factors  as  net  cost/savings  and  return  on  investment.  The  nondollar  benefits 
and  liabilities  addressed  in  this  task  (Task  6)  will  not  be  included  in  the 
financial  analysis.  However,  the  results  of  the  two  distinct  decision  pro- 
cesses will  be  presented  to  the  decision  maker  as  outlined  in  Task  8 (Make 
Accept/Reject/Study  Decision).  The  major  subtasks  within  Task  6 are  out- 
lined below. 

SUBTASK 

1.  Assemble  the  data  from  the  preceding  tasks  including 
the  newly  acquired  range-of-effect  and  benefits  data 
from  Task  5. 

2.  Prepare  appropriate  cases  and  cost  model  input  data 
for  each  decision  tree  outcome. 


REFERENCE 

06.01.01 

06.02.01 
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3.  Run  cost  model  for  each  case  to  determine  dollar  value  06.03.01 

of  each  decision  tree  outcome. 

4.  Determine  equivalent  dollar  value  for  each  of  the  benefit/  06.04.01 
liability  patterns. 


5.  Refine  the  success  probability  estimates  considering  any  04.02.01 
additional  risks  from  any  extended  application  indicated  04.04.01 
from  Task  5. 

6.  Develop  new  decision  tree  assessment  diagram  using  the  06.06.01 

parameters  developed  in  the  preceding  subtasks;  then 

determine  new  decision  values. 

7.  Perform  sensitivity  analysis  of  decision  path  to  06.07.01 

selected  variables. 

8.  Create  tickler  file  entry  identifying  criteria  for  re-  04.06.01 

assessment  of  innovation  if,  at  this  point,  the  proposal 

does  not  appear  feasible. 


* 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  ASSEMBLE  DATA 


DESCRIPTION 

The  cost-benefits  analysis  further  develops  the  decision  tree  assessment  for- 
mat from  Task  4.  These  data,  along  with  the  full  range  of  entities  (courses, 
jobs,  equipment,  etc.)  and  benefit  characteristics  identified  in  Task  5,  will 
form  the  basis  for  the  refined  cost-benefits  analysis  of  Task  6. 

PROCEDURE 

1.  Obtain  the  final  decision  tree  prepared  in  04.05  along  with  rationale 
supporting  the  separation  of  costs/savings  by  outcome  and  the  probabili- 
ties assigned  to  implementation  and  user  acceptance  success. 

2.  Obtain  the  final  full  range  of  potential  applications  for  the  innovation 
from  Task  5.  These  will  include,  where  appropriate: 

• Course  numbers  (CDP) 

• Job  designators  (Rate,  NEC,  NOBC,  etc.) 

• Equipment  types 

• Student  types 

• Cost  elements 

• Taxonomic  search  parameters 

• Other  directly  costable  elements. 


3.  Obtain  the  allocation  rules  which  identify  how  the  pattern  of  benefits/ 
liabilities  is  to  be  apportioned  across  each  of  the  entities  selected 
as  being  potentially  impacted  by  the  proposed  innovation. 

COMMENT:  Typical  allocation  rules  might  be:  1)  10%  of  the  laboratory  time 

in  course  12AB  will  be  saved;  2)  one  hour  of  maintenance  per  month  will  be 
saved  on  the  XXXX  task  trainer;  3)  three  weeks  of  OJT  time  for  ET2's  and 
ET3's  with  the  following  NEC's  assigned  to  shipboard  duty  will  be  saved. 

4.  Obtain  the  full  benefit/liabilities  pattern  developed  in  Task  5 which 
will  include: 

• Specific  relevant  variables  impacted  by  the  application  of  the 
innovation;  e.g.,  time  to  train,  attrition,  etc. 


III-166 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 

* 

06  01  02 

• A figure  indicating  the  relative  importance  of  each  variable  as 
compared  to  all  relevant  ones;  e.g.,  crucial  importance  = 10, 
trivial  importance  = 1,  etc. 

• Magnitude  and  direction  of  change  in  each  relevant  variable;  e.g., 
'attrition  reduced  from  20%  to  18%  or  attrition  reduced  10%. 

t A confidence  range  for  the  estimate  of  each  relevant  variable;  e.g., 
under  the  most  adverse  conditions,  the  most  likely  estimate  of 
attrition  (18%)  will  be  no  greater  than  20%  and  under  the  most 
favorable  conditions,  it  could  go  as  low  as  15%,  or  the  most  likely 
attrition  reduction  estimate  of  10%  may  vary  from  0%  (worst  case) 
to  25%  (best  case). 

5.  Derive  the  expected  value  of  the  relevant  variables.  Since  the  variables 
will  generally  be  bounded  at  both  ends  by  practical  limits  and  these 
limits  are  expressed  by  the  confidence  range  identified  as  a part  of 
the  benefits/liabilities  pattern  developed  in  Task  5,  a useful  estimate 
of  the  expected  value  (vj:)  is  the  PERT  statistic:* 

V£  * p + 4m  + o 


Where:  = expected  value  of  the  relevant  variable 

p = most  pessimistic  value  of  the  variable 
m = most  likely  estimate  of  the  variable 
o = most  optimistic  estimate  of  the  variable. 

The  value  derived  will  be  used  as  the  cost  model  and  the  decision  tree  inputs. 

COMMENT:  The  assessor  may  want  to  review  06.07  (Perform  Sensitivity  Analysis) 
to  become  familiar  with  the  further  use  of  the  expected  value  and  confidence 
range.  Also,  some  degree  of  caution  should  be  used  in  this  expected  value 
estimate,  especially  when  the  distribution  using  p,  m,  and  o is  highly  skewed. 

If  it  appears  that  the  decision  will  be  sensitive  to  certain  variables,  more 
systematic  methods  are  available  and  may  be  appropriate  for  obtaining  the  ex- 
pected value  and  distribution  for  the  variables  (see  Handbook  for  Decision 
Analysis  - Chapter  9). 


*' Hahn,  Gerald  0.  and  Samuel  S.  Shapiro,  STATISTICAL  MODELS  IN  ENGINEERING, 
John  Wiley  & Sons,  Inc.,  New  York,  1967,  p.  133.' 

Battersby,  A.,  NETWORK  ANALYSIS  FOR  PLANNING  AND  SCHEDULING,  St.  Martin 
Press,  Inc.,  New  York,  1964,  p.  174. 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  PREPARE  COST  MODEL  CASES  AND  INPUT  DATA 


DESCRIPTION 

Four  basic  outcomes  (A  through  D)  have  been  defined  on  the  decision  tree 
assessment  diagram.  These  range  from  acceptance,  successful  implementation, 
and  user  acceptance  to  rejection  of  the  proposal.  (This  is  not  necessarily 
the  range  considering  the  net  of  costs  and  savings.)  An  alternative  case  must 
be  established  for  each  of  the  possible  outcomes  which,  in  the  case  of  failure, 
means  the  formulation  of  a recovery  strategy  to  which  certain  costs  will  be 
attached.  Each  defined  case  must  have  the  appropriate  cost  model  and  input 
variables  specified. 

PROCEDURE 

1.  Develop  basic  scenarios  for  each  of  the  possible  decision  tree  outcomes. 

In  the  case  where  the  proposed  innovation  is  accepted,  implemented,  and 
used  over  an  extended  planning  horizon,  basic  development,  installation, 
and  operational  factors  will  be  considered.  If  contingency  plans  will 
involve  identifiable  cost  elements,  these  should  be  included  in  the 
scenarios. 

The  scenario  for  outcome  B,  where  the  innovation  is  successfully  imple- 
mented but  fails  to  gain  user  acceptance  over  a reasonable  period  of 
time,  might  contain  the  following  assumptions. 

• Initial  planning  is  the  same  as  outcome  A. 

• A two-year  period  will  be  considered  as  a norm  to  determine  acceptance. 

• The  measures  of  user  acceptance  planned  for  outcome  A will  be  incorpo- 
rated, except  that  measurement  cost  will  increase  by  1 0%  due  to  in- 
creased sampling  as  measurements  move  toward  the  control  limits. 

• If  rejection  is  indicated  at  the  end  of  two  years,  the  baseline  plan 
will  be  used  for  recovery  purposes. 

• Equipments  purchased  under  the  proposed  initial  plan  will  have  no  re- 
maining value  due  to  their  specialized  character. 

A similar  scenario  should  be  developed  for  outcome  C.  Outcome  D,  to  reject 
the  innovation,  will  generally  be  considered  neutral. 


_ 
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2.  Prepare  cost  model  input  data  sheets  specifying  the  values  of  each  variable 
under  each  outcome  outlined  by  the  preceding  scenarios.  Input  data  sheets 
should  be  prepared  for  each  model  appropriate  to  the  costing  exercise;  i.e., 
the  training  cost  model  will  accept,  for  example,  the  following  factors. 

t Technical  factors 

Annual  requirements  for  course  graduates 

Attri ti on  rate 

Course  length 

Student  recycle  rate  (percent  of  students  set  back) 

Average  recycle  time  (time  added  by  being  set  back) 

Average  equipment  downtime 

Instructor/student  ratio 

Administrative/student  ratio 

Equipment/student  ratio 

Facilities  square  feet/student 

Facilities  square  feet/instructors 

Facilities  square  feet/administrative 

Instructional  materials  development  ratio 

Amount  of  course  material  requiring  new  development 

Percent  annual  course  maintenance  requirements 

Supplies/student 

Miscellaneous  requirements/student, 
t Cost  Factors 

Student  salary 
Student  travel 
Instructor  salary 
Administrative  salary 
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Equipment  unit  costs 
Life  of  equipment 
Purchase  policy 
Depreciation  policy 

Operation  and  maintenance  costs  for  equipment 

Cost/square  foot  of  space 

Instructional  materials  development  cost 

Value  of  instructional  materials  at  end  of  planning  period 

Supply  costs/student  and  student  position 

Miscellaneous  costs/student  and  student  position. 

The  job  model  may  be  developed  to  accept,  for  example,  the  following 
factors . 

• Annual  incumbents  affected 

• Job  classification  (Rate,  NEC,  etc.) 

• Years  service 

• Time  in  grade 

• Support  personnel  ratio 

• Job  support  space  (facilities  square  feet) 

t Tools  unit  cost 

• Equipment  unit  cost 

• Salary  rate  (incumbent  and  support) 

• Recruiting/induction  costs 

• School  training  costs 

• Transportation  costs 
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• Operational  time  utilized 

• Unutilized  time  cost 

V 

• Error  rate 

• Error  cost. 

The  equipment  model  may  be  developed  to  accept,  for  example,  essentially 
the  equipment  related  items  within  the  training  cost  model. 

• Numbers  of  equipment 

• Unit  cost  of  equipment 

• Operations  and  maintenance  cost 

• Life  of  equipment 

• Purchase  policy 

§ Depreciation  policy 

• Utilization. 

COMMENT:  Each  cost  model  when  developed  will  have  specific  formats  for  the 

input  data.  The  approach  planned  for  the  training  cost  model  is  to  define 
absolute  rather  than  incremental  cost  factors.  This  will  require  that  the 
input  data  for  the  baseline  plan  be  defined  also,  and  the  incremental  value 
of  output  variables  must  be  obtained  by  taking  the  difference  between  the 
two  runs.  Also,  costs  should  be  classified  along  an  additional  dimension  to 
avoid  any  major  restructuring  in  Task  7.  The  cost  categories  for  grouping 
data  elements  are  1)  R&D,  2)  investment,  and  3)  recurring. 

3.  Ensure  that  the  input  variables  reflect  the  true  "opportunity  cost"  re- 
sulting from  the  proposal  application.  For  example,  if  three  weeks  of  OJT 
time  were  defined  as  being  saved  by  the  proposed  innovation,  the  full 
salary  costs  or  the  full  three  weeks  should  only  be  taken  as  a savings  if 
there  is  an  alternate  opportunity  to  use  the  time  saved  at  the  value 
assigned.  This  should  be  considered  over  the  entire  planning  period 
under  consideration,  since  while  there  may  be  no  value  for  the  time 
initially,  additional  workload  in  subsequent  years  may  have  value.  The 
value  for  the  entire  planning  period  should  be  factored  into  the  input 
variables. 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  RUN  APPROPRIATE  COST  MODELS 


DESCRIPTION 

The  following  outlines  the  basic  steps  in  making  the  cost/savings  calculation. 

While  these  may  be  carried  out  manually,  generally  the  input  data  from  the 

preceding  subtasks  will  be  processed  using  some  form  of  Automated  Data 

Processing  (ADP). 

PROCEDURE 

1.  Review  input  data  sheets  to  ensure  relevant  variables  have  been  incorporated, 
that  they  are  adjusted  to  reflect  the  true  economic  value  (opportunity  cost) 
achieved  by  applying  the  innovation,  that  is,  the  value  sacrificed  by  the 
resource  not  being  available  for  some  alternative  use.  If,  for  example,  a 
certain  amount  of  building  space  is  unused,  then  it  has  essentially  no 
value.  If,  however,  there  were  a requirement  for  the  space  from  some  other 
source,  and  that  source  valued  the  space  at  its  existing  cost,  then  the 

full  cost  would  be  used. 

2.  Training  Cost  Model.  The  development  of  training  costs  will  generally  be 
performed  on  a course  by  course,  or  program,  basis.  The  major  cost  cate- 
gories and  the  general  procedural  steps  for  calculating  these  categories 
are  outlined  on  the  following  pages. 

• Determine  Student  Requirements.  Given  the  annual  requirements  for 
graduates,  calculate  annual  student  input  and  student  load  considering 
the  attrition  rate,  recycle  rate,  average  recycle  time,  and  course 
length. 

• Determine  Student  Position  Requirements.  Given  the  student  average-on- 
board (AOB),  calculate  the  number  of  student  positions  required  con- 
sidering the  time  a student  position  may  be  unavailable  (e.g.,  for 
maintenance)  and  the  percent  of  extra  student  positions  required  (e.g., 
for  queueing). 

• Determine  Personnel  Requirements.  Given  the  number  of  student  positions, 
calculate  instructor  personnel  required  considering  the  instructor  to 
student  position  ratio,  and  calculate  administrative  personnel  required 
considering  the  administrator  to  student  position  ratio. 
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• Determine  Equipment  Required.  Given  the  number  of  student  positions, 
calculate  equipments  require’  ’ ’to  student 


its  life  in  relation  to  the  r „ f the 

particular  purchase  policy  selected;  e.g.,  purchase  only  to  80%  of  the 
maximum  requirement  over  the  planning  period. 

• Determine  Facilities  Required.  Given  the  number  of  student  positions, 
calculate  total  facilities  required  considering  the  square  feet  per 
student  position  for  students,  instructors,  and  administrators. 

• Determine  Instructional  Materials  Development/Maintenance  Required. 
Calculate  the  amount  of  time  for  developing  initial  course  materials 
considering  the  course  length,  the  percent  of  the  course  requiring 
development,  and  the  development  ratio.  Given  the  amount  of  time  for 
developing  initial  course  materials,  calculate  the  annual  maintenance 
requirements  considering  the  percent  of  initial  development  time  to 
be  expended  annually  for  maintenance  or  course  update. 

• Determine  Supplies  Required.  Given  the  number  of  students  or  number 
of  student  positions,  calculate  supplies  required  considering  the 
supplies  per  student  ratio  or  the  supplies  per  student  position  ratio, 
whichever  is  appropriate. 

• Determine  Miscellaneous  Required.  Given  the  number  of  students  or 
number  of  student  positions,  calculate  miscellaneous  required  con- 
sidering the  miscellaneous  per  student  ratio  or  the  miscellaneous  per 
student  position  ratio,  whichever  is  appropriate. 

• Determine  Training  Costs.  Training  costs  are  calculated  by  taking 
each  of  the  preceding  technical  requirements  times  its  unit  cost. 

Student  costs  consider  salary  costs  per  student  year  (AOB),  student 
travel  to  and  from  school,  as  well  as  any  travel  while  in  training. 

Personnel  costs  consider  instructor  salaries  and  administrative 
salaries. 

New  equipment  costs  consider  the  individual  equipment  purchase 
cost  and  annual  maintenance  cost.  Based  upon  the  defined  deprecia- 
tion rate,  the  life  of  the  equipment,  and  the  length  of  the  planning 
period,  equipment  may  have  a remaining  value  which  will  be  sub- 
tracted from  costs.  (The  assumption  here  is  that  the  equipment  can 
be  sold  or  put  to  an  alternative  use  which  is  equal  to  its  remaining 
value.)  Used  equipment  valuation  should  be  based  upon  its  opportunity 
cost,  that  is,  its  value  in  alternative  uses. 


position  ratio.  The  actual 


function  of 
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Facilities  costs  consider  the  total  square  footage  requirements 
times  a cost  per  square  foot  factor. 

Instructional  materials  costs  consider  the  total  development  and 
maintenance  time  requirements  and  the  cost  per  unit  of  time. 
Instructional  materials,  like  equipment,  may  have  some  remaining 
value  based  upon  some  alternate  use  at  the  end  of  the  planning 
period.  Considering  a percentage  factor  for  end  of  life  value, 
the  offset  against  total  cost  can  be  calculated  to  arrive  at  a 
net  cost  of  instructional  materials. 

Supplies  and  miscellaneous  costs  can  be  calculated  from  the  unit 
cost  factors. 

• Aggregate  Costs.  Sum  the  costs  on  an  annual  basis  over  the  planning 
period.  A total  (nondiscounted)  cost  is  obtained  by  sunning  the 
annual  requirements.  A more  realistic  economic  value  which  considers 
the  time  value  of  money  is  an  overall  value  which  has  been  discounted 
at  a rate  representing  the  annual  cost  of  capital.  The  annual  dis- 
counting should  be  performed  and  the  discounted  annual  costs  summed 
to  obtain  a present  value  cost  figure  for  the  program. 

• Develop  Unit  Factors.  It  is  often  valuable  for  comparative  purposes 
to  calculate  the  program  cost  per  graduate,  cost  per  student  input, 
cost  per  student  position,  or  cost  per  student  day  of  training.  This 
can  be  done  with  both  the  nondiscounted  or  discounted  factors  and 
displayed  on  an  annual  and  total  program  basis. 

3.  Job  Cost  Model.  In  general,  the  determination  of  seemingly  quantifiable 
job  effects  will  start  with  experts,  knowledgeable  of  a specific  job  (or 
task)  situation,  predicting  training  variations  or  modifications  upon  the 
appropriate  job  variable;  e.g.,  performance  time,  error  rates,  support 
requirements,  etc.  For  life  cycle  planning  purposes,  generally  done  well 
in  advance  of  implementation,  savings  of  personnel  will  reflect  in  a re- 
duction of  a number  of  costs  including  base  pay,  hazardous  duty  pay, 
transportation  costs,  retirement  benefits,  etc.  Navy  Military  Manpower 
Billet  Cost  Data  for  Life  Cycle  Planning  Purposes  (NAVPERS  15163)  provides 
costing  information  which  can  be  applied  once  an  estimate  of  the  number 
of  personnel  affected  has  been  made.  Again,  the  difficulty  is  in  evaluating 
the  opportunity  cost  for  time  saved  at  a task  or  duty  level.  The  question 
as  to  whether  time  saved  can  be  put  to  some  alternative  use  with  value 
approximating  the  factors  in  the  Billet  Cost  Data  is  more  easily  answered 
when  preparing  manning  level  estimates  well  in  advance  of  implementation 
than  it  is  when  performance  times  on  existing  jobs  are  being  affected  by 
an  innovation.  \ 
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Some  work  has  been  accomplished  in  estimating  the  cost  of  on-the-job  train- 
ing which  could  be  one  of  the  more  frequently  impacted  variables  in  the 
assessment  of  training  innovations.  While  a general  methodology  has  been 
developed  for  making  OJT  cost  assessments,  models  would  have  to  be 
developed  for  each  job  type  (rating,  etc.)  potentially  impacted.  The 
elements  of  cost  identified  in  an  Air  Force  Study*  are: 

• Trainee  time 

t Instructor  time 

• Delved  entry  time 

• Records  management  time 

• Remedial  training  time 

• Equipment  and  material. 

Various  models  for  estimating  costs  in  each  category  were  examined  and  an 
appropriate  one  selected  for  predicting  each  cost  element.  If  specific 
models  and  data  such  as  those  discussed  above  are  available  and  appropriate, 
they  should  be  used,  however,  a general  procedural  approach  for  combining 
the  data  input  factors  specified  in  06.02  is  as  follows. 

• Determine  Personnel  Costs.  Given  the  number  of  annual  incumbents 
affected,  salary  rates  can  be  identified  based  upon  job  classification, 
years  service,  and  time-in-grade  factors.  The  operational  time  utilized 
and  unutilized  time  cost  become  the  basis  for  valuing  the  time  saved  or 
additional  time  used  as  a result  of  the  proposed  innovation.  Overhead 
or  support  personnel  can  be  determined  from  the  support  personnel  ratio 
and  total  cost  determined  using  the  salary  rate.  Depending  upon  the 
nature  of  the  innovation's  impact  over  the  planning  period,  total 
personnel  costs  could  include  the  additional  costs  of  recruiting/induc- 
tion, school  training,  and  transportation. 

• Determine  Facilities  Costs.  Job  support  space  required  will  be  identi- 
fied and  cost  determined  using  the  cost/square  foot  factor. 

• Determine  Job  Performance  Support  Costs.  Certain  jobs  require  special- 
ized tools  or  equipment  which  can  be  costed  using  unit  cost  factors  for 
each. 

Earners,  B.N.,  Dunham,  A.D.  and  Nordhauser,  F. , 1974.  The  Development  of  a 
Methodology  for  Estimating  the  Cost  of  Air  Force  On-The-Job  Training.  Report 
No.  AFHRL-TR-74-34,  Contract  F41609-72-C-0048,  AFHRL,  Brooks  AFB,  Texas. 


III-175 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
06  03  05 


• Determine  Performance  Change  Costs.  Some  innovations  may  be  modifying 
the  rate  of  operational  errors  made  by  the  job  incumbent.  The  error 
rate  and  cost  per  error  factors  are  intended  to  permit  a cost  estimate 
of  this  job  attribute. 

• Aggregate  Costs.  Sum  the  costs  on  an  annual  basis  over  the  planning 
period.  A total  (nondiscounted)  cost  is  obtained  by  summing  the 
annual  requirements.  A more  realistic  economic  value  which  considers 
the  time  value  of  money  is  an  overall  value  which  has  been  discounted 
at  a rate  representing  the  annual  cost  of  capital.  The  annual  dis- 
counting should  be  performed  and  the  discounted  annual  costs  sunned 
to  obtain  a present  value  cost  figure  for  the  program. 

t Develop  Unit  Factors.  Comparisons  of  alternatives  and  before/after 
situations  can  be  made  if  certain  types  of  unit  factors  are  developed 
such  as,  cost  per  job  hour,  cost  per  job  hour  utilized,  error  cost 
per  incumbent,  support  costs  per  incumbent,  etc. 

4.  Equipment  Cost  Model.  Equipment  costs  are  derived  by  combining  the  factors 

in  the  following  general  manner. 

• Determine  Numbers  of  Equipment  Affected.  When  an  innovation  causes 
equipment  to  be  either  displaced  or  procured,  the  numbers,  by  type, 
will  be  specified  from  Task  5. 

t Identify  Related  Resource  Impacts.  The  displacement  or  procurement  of 
equipment  may  affect  other  costable  items  such  as: 

Refurbishing  of  facilities  costs 

Transportation 

Inventories  (e.g.,  spare  parts) 

Maintenance 
Working  funds 
Supplies 
Uti 1 i ti es . 

• Identify  Unit  Costs.  For  equipments  and  other  costable  items,  determine 
the  unit  costs  from  data  bases  or  other  sources. 

• Equipment  Purchase  Costs.  Equipment  life  (actual,  not  accounting)  must 
be  considered  to  determine  the  number  of  procurements  to  be  made  over 
the  planning  cycle  as  well  as  the  potential  cash  flows  resulting  from 
an  end-of-life  equipment  value.  Purchase  and  depreciation  policies 
will  affect  investment  and  other  cash  flows. 


III-176 


TAEG  REPORT  NO.  12-3 


TASK  SUBTASK  PAGE 
06  03  06 


t Equipment  Utilization.  Utilization  would  be  an  important  considera- 
tion in  arriving  at  an  alternative  use  cost  when  displacing  or  using 
existing  equipment.  When  equipment  is  already  owned,  its  economic 
costs  are  approximated  by  estimating  its  value  in  alternative  uses. 

5.  Develop  the  incremental  cost/savings  value  for  each  of  the  model  cases 
run.  This  will  be  the  difference  between  the  baseline  or  existing  system 
and  the  proposed  systems  for  each  decision  tree  outcome,  A through  D.  If 
the  innovation  requires  use  of  more  than  one  model,  the  incremental  values 
of  each  should  be  summed  and  assigned  to  the  respective  outcome.  Be  sure 
that  paths  (2),  etc.,  contain  the  implementation  costs  identified  in  04.03 
(Define  Implementation  Projects  and  Costs).  These  data  will  be  used  in 
Subtask  06.06  (Develop  Decision  Tree  Assessment)  and  in  Task  7 (Perform 
Financial  Analysis). 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  DETERMINE  BENEFIT/LIABILITY  VALUES* 


DESCRIPTION 

The  preceding  sub tasks  have  dealt  with  the  tangible  values  of  costs  and 
savings.  Sometimes  intangible  factors  may  have  a significant  effect  upon 
the  path  a decision  ultimately  takes.  The  purpose  of  this  subtask  is  to 
systematically  develop  value  or  utility  for  the  otherwise  nonquantifiable 
benefits  and  liabilities  which  result  from  application  of  the  innovation. 

The  value  obtained  from  this  procedure  will  be  expressed  in  equivalent 
dollars  and  will  be  developed  for  each  of  the  decision  tree  outcomes  so  that 
the  values  can  be  added  to  the  previously  developed  costs  and/or  savings. 

PROCEDURE 

1.  Obtain  the  data  assembled  in  06.01  for  the  nonquantified  benefits  and 
liabilities.  Relevant  variables  may  include  process  or  procedural  effec- 
tiveness, attitudes,  morale,  communications,  safety,  security,  environ- 
mental quality,  manageability,  controllability.  Magnitude  estimates  and 
direction  of  change,  relative  importance,  and  confidence  range  will  have 
been  identified  for  each  variable. 

2.  Assess  the  relative  effect  of  a single  benefit/liability  dimension  for 
each  decision  tree  outcome.  Rank  the  outcomes  according  to  their  rela- 
tive value  on  a format  as  shown  in  Figure  6.4.1.  Remember,  only  a single 
factor  is  being  considered  at  this  time;  for  example,  student  morale.  It 
is  possible  that  this  variable  could  be  ranked  in  several  ways  depending 
upon  the  type  of  innovation.  If  the  intended  result  of  the  innovation 
significantly  improves  student  morale,  then  to  accept,  implement,  and  have 
user  acceptance  might  be  ranked  first,  while  to  accept,  implement,  and 
have  user  rejection  (outcome  B)  might  be  ranked  last.  On  the  other  hand, 
if  student  morale  is  negatively  impacted  (a  liability  of  the  proposed 
innovation),  then  to  reject  outright  (outcome  D)  might  be  ranked  first 
while  to  accept,  implement,  and  have  user  acceptance  (outcome  A)  might  be 
ranked  last. 

3.  Develop  the  weights  for  each  of  the  ranked  outcomes  on  a scale  of  0 to  100. 
The  first  ranked  outcome  is  assigned  a value  of  100  while  the  last  ranked 
outcome  is  assigned  a value  of  0.  Determine  which  outcome  is  neutral.  By 
determining  how  much  better  or  how  much  worse  one  outcome  is  in  relation 

to  another,  each  outcome  can  be  assigned  an  intermediate  value  between  0 
and  100.  Figure  6.4.2  shows  a completed  sheet  where  each  outcome  has  been 


*The  approach  outlined  here  was  adapted  from  the  "Handbook  for  Decision 
Analysis",  Decision  and  Designs,  Inc.,  ONR  Contract  N0001 4-73C-01 49 , and 
should  be  reviewed  prior  to  this  phase  of  the  assessment. 
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assigned  a value.  The  rationale  for  the  weights  has  also  been  included. 
The  consistency  of  estimates  can  be  checked  by  standard  gambles.  For 
example,  if  A = 100,  B = 50,  and  D = 0,  does  B approximately  equal  a 
gamble  between  A and  D each  having  a 50£  probability  of  occurring?  If 
this  is  a reasonable  approximation,  then  the  values  are  consistent. 

4.  Select  another  variable  and  repeat  the  analysis  of  steps  2 and  3. 


TAEG  REPORT  NO.  12-3 


OUTCOME  ANALYSIS  - BENEFIT  RANKING  AND  VALUATION 


RANK 


OUTCOME 

LETTER 


OUTCOME  NAME 


VALUE  SCORE 

r~  - - 100 


90 


1 


80 


2 


70 


60 


3 


60 


40 


4 


30 


20 


10 


0 


FIGURE  6.4.1.  BENEFIT  OUTCOME  ANALYSIS  FORMAT 
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BENEFIT:  STUDENT  MORALE 
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FIGURE  6.4.2.  BENEFIT  OUTCOME  ANALYSIS  FORMAT  (COMPLETED  EXAMPLE) 
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TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  DEVELOP  DECISION  TREE  ASSESSMENT 


DESCRIPTION 

The  purpose  of  this  subtask  is  to  combine  the  results  of  the  cost  model  runs 
and  the  noncost  benefits  analysis  by  their  relative  importance  weights  to 
obtain  a total  value  for  each  decision  tree  outcome  in  equivalent  dollars. 

The  refined  success  probabilities  and  the  equivalent  dollar  values  are 
entered  into  the  decision  tree  assessment  format  and  folded  back  to  find  the 
decision  path  with  the  highest  expected  value. 

PROCEDURE 

1.  Gather  the  results  of  the  cost  model  runs  from  06.03,  the  benefit/ 
liabilities  value  scales  from  06.04,  the  refined  success  probabilities 
from  06.05  (repeat  of  04.02  and  04.04),  and  the  importance  values  for 
each  variable  assembled  in  06.01.  The  data  will  be  documented  on  a 
Benefit  Outcome  Summary  Format  as  shown  in  Figure  6.6.1. 

2.  • Develop  equivalent  dollars  for  each  relevant  variable  by  weighting  each 

variable's  scale  by  the  relative  importance  factor.  This  procedure 
involves : 

a.  Indicating  the  cost/savings  value  for  each  outcome.  (See  the  example 
of  the  Training  ACTUAL  DOLLARS  in  Figure  6.6.2.) 

b.  Determining  the  range  of  the  scale  for  all  outcomes.  (In  Figure  6.6.2, 
ACTUAL  DOLLARS  go  from  +6.0  to  -3.0,  therefore,  the  range  is  9.0.) 

c.  Establishing  the  relative  range  of  a new  variable  by  the  ratio  of  the 
new  variable  importance  to  the  original  dollar  importance.  (In 
Figure  6.6.2,  the  ratio  of  STUDENT  MORALE  to  ACTUAL  DOLLARS  is  8:5  or 
1.33.) 

d.  Apportioning  the  0-100  scale,  for  the  new  variable  obtained  from  the 
Benefit  Outcome  Analysis,  across  the  new  range.  (In  Figure  6.6.2,  the 
new  range  is  1.33  X 9.0  or  12.0,  therefore,  the  STUDENT  MORALE  variable 
in  the  0-100  scale  from  Figure  6.4.2  is  spread  across  this  new  range. 
Note  that  the  range  goes  from  +2.4  to  -9.6  (12.0)  because  of  the  rela- 
tionship of  the  zero  outcome  (D)  to  the  entire  scale.) 
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3.  Display  the  range  of  values  for  all  outcomes  and  for  each  variable  on  the 
Benefit  Outcome  Summary  and  summarize  the  total  for  each  outcome  as  shown 
on  Figure  6.6.3. 

COMMENT:  A major  assumption  in  this  summarization  is  that  the  utility 

attributes  (e.g.,  ACTUAL  DOLLARS  and  STUDENT  MORALE)  can  be  added  together. 
Requirements  (not  discussed  here)  which  include  utility  independence,  pairwise 
preferential  independence  and  pairwise  marginal ity,  determine  if  the  additive 
form  holds.*  In  general,  it  appears  that  most  attributes  which  will  be 
combined  in  the  training  domain  will  satisfy  the  requirements  for  additivity. 
Certain  tests  can  be  made  as  outlined  in  the  referenced  literature  if  the 
assessor  is  concerned  about  meeting  these  requirements. 

4.  Complete  the  decision  tree  assessment  diagram  by  entering  the  refined 
success  probabilities  at  the  event  nodes  and  the  values  for  each  outcome 
at  the  end  of  each  decision  path. 

5.  Fold  back  the  decision  tree  as  outlined  in  the  procedure  of  04.05.  Figure 
6.6.4  provides  an  example  of  a completed  decision  tree. 

6.  Review  the  decision  variable  results  for  reasonability. 

COMMENT:  The  decision  tree  in  the  form  presented  does  not  include  a considera- 

tion of  sequencing  projects  in  a way  that  the  decision  to  proceed  is  a function 
of,  for  example,  successful  completion  of  project  A followed  by  successful 
completion  of  project  B.  If  it  is  thought  that  this  decision  structure  will 
exist  and  may  have  an  effect  upon  the  decision  outcome,  then  the  assessor  may 
want  to  make  a decision  tree  modification  to  reflect  the  decision  situation 
more  accurately. 

The  process  described  above  is  only  one  facet  of  the  decision  process,  the  other 
is  the  financial  analysis.  However,  if  the  decision  indicated  as  a result  of 
this  process  does  not  appear  reasonable  from  a global  viewpoint,  the  individual 
decision  components  should  be  rechecked  for  reasonability. 


*Wi 1 1 i am  C.  Giauque.  1975  Decision  Making  in  the  Naval  Education  and 
Training  System.  TAEG. 
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RELEVANT 

VARIABLE 

RELATIVE 

IMPORTANCE 

A 

B 

C 

D 

COMMENT 

ACTUAL 

DOLLARS 

(Training) 

6 

6.0 

2.0 

1.0 

0 

Cost/savings  data 
from  running  Training 
Cost  Model 

STUDENT 

MORALE 

8 

100 

0 

60 

80 

Benefit  outcome 
analysis  format 
Figure  6.4.1 
Example 

(ADJUSTED)* 

1.6 

-6.4 

-1.6 

0> 

NOTE:  The  STUDENT  MORALE  variable  is  8:6  or  1.33  times  as  important  as  the 

ACTUAL  DOLLARS,  and  outcome  D is  neutral  or  zero.  Since  the  ACTUAL 
DOLLAR  range  is  +6.0  to  -0.0  or  6.0,  and  STUDENT  MORALE  is  1.33  times 
as  important,  the  total  STUDENT  MORALE  range  is  1.33  X 6.0  = 8.0. 
Outcome  A is  20%  of  the  total  range  in  the  positive  direction,  there- 
fore, outcome  A is  1.6.  Outcome  B is  80%  of  the  range  in  a negative 
direction,  therefore,  outcome  B is  -6.4. 


FIGURE  6.6.2.  BENEFIT  OUTCOME  SUMMARY 
(SAMPLE  DATA) 


1 11-185 


TAEG  REPORT  NO,  12-3 


OUTCOME  ANALYSIS  - BENEFIT  UTILITY  ASSIGNMENT 


ASSESSMENT 

DIMENSIONS 


to 

Z 

o 


5 

to 

cc 

< 

-J 

-J 

§ 

K 

Z 

Ui 

-J 

< 

> 

5 

0 

LU 

1 

00 


a 

H 

D 

O 


UL 

O 

ui 

D 

-J 

< 

> 


OUTCOME 

SUMMARY 


FIGURE  6.6.3  BENEFIT  OUTCOME  SUMMARY  (EXAMPLE) 
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FIGURE  6.6.4.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 

(EXAMPLE  USING  SAMPLE  DATA) 


A SAVING* 'BENE  FITS 


SAVINGS  T 

ACCRUt 


■ COSTS  • POSSIBLE 
LIABILITIES  RESULT 


C COSTS  ACCRUE 


D NEUTRAL  OUTCOME 

A SAVINGS/BENE  PITS 
ACCRUE 

B COSTS  POSSIBLE 
LIABILITIES  RESULT 

C COSTS  ACCRUE 


0 NEUTRAL  OUTCOME 

A SAV1NGS/BENEPITS 
ACCRUE 

b com  possible 

LIABILITIES  RESULT 
C COSTS  ACCRUE 
O NEUTRAL  OUTCOME 


1 


TAEG  REPORT  NO.  12-3 


TAEG  REPORT  NO.  12-3 


TASK 

SUBTASK 

PAGE 

06 

07 

01 

TASK:  PERFORM  COST-BENEFITS  ANALYSIS 


SUBTASK:  PERFORM  SENSITIVITY  ANALYSIS 


DESCRIPTION 

The  assessor  may  be  faced  with  a decision  in  which  there  is  very  little 
difference  in  the  decision  variable  values  for  the  alternative  choice;  e.g., 
to  accept  without  continued  study  versus  to  accept  with  continued  study.  The 
purpose  of  this  subtask  is  to  present  the  assessor  with  an  insight  into  the 
variables  which  might  change  the  decision  which  is  presently  indicated.  This 
is  accomplished  by  determining  which  variables;  e.g.,  estimates  of  success 
probabilities,  relevant  costable  and  noncostable  variables,  or  measures  of 
relative  importance,  may  cause  a change  in  the  decision  considering  the  con- 
fidence range  that  the  estimator  originally  had  in  those  particular  variables. 
The  results  of  this  analysis  may  lead  to  the  gathering  of  additional  informa- 
tion or  specific  variables  to  which  the  decision  is  sensitive  so  that  the 
confidence  in  the  estimate  of  their  value  can  be  improved. 

PROCEDURE 

1.  Review  the  sensitivity  analysis  diagrams  for  the  models  used  to  develop 
the  cost  factors  to  determine  if  any  of  the  variables  affected  by  the 
proposed  innovation  are  particularly  sensitive.  Figures  6.7.2  through 
6.7.7,  are  sample  sensitivity  graphs  for  the  Training  Cost  Model. 

2.  Identify  the  variables  to  be  inputted  to  the  sensitivity  analysis  program 
which  will  perform  the  following  analytic  steps. 

a.  Determine  if  the  inputted  variables  cause  a change  in  the  indicated 
decision  across  the  expressed  confidence  range. 

b.  If  no  confidence  range  has  been  expressed;  e.g.,  where  the  relative 
importance  variable  is  concerned,  the  variable  values  for  which  the 
decision  changes  will  be  calculated. 

c.  A sensitivity  and  probability  that  the  indicated  decision  will  change, 
will  be  calculated  for  each  variable  that  causes  a change  within  the 
confidence  range. 

d.  The  break-even  point  value  for  each  variable  from  b.  and  c. , above, 
will  be  calculated. 

e.  An  output  report  as  shown  in  Figure  6.7.1  will  be  generated  for  the 
assessor  to  review. 
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COMMENT:  The  probability  generated  should  be  applied  with  caution  since  some 

major  assumptions  are  involved  in  its  calculation.  First,  as  with  the  calcula 
tion  of  the  mean  or  expected  value,  an  approximation  of  the  beta  distribution 
is  assumed,  therefore,  the  standard  deviation  is  calculated  by: 

S = Oi£  f 

6 

Where:  S = standard  deviation 

o = optimistic  estimate 
p = pessimistic  estimate. 

A Z-value  is  calculated,  assuming  the  beta  approximates  the  normal  distribu- 
tion by  the  formula: 

Z = b-V£ 

~S 

Where:  Z = Z-value 

b = break-even  value 

V£=  expected  value  (mean  - see  06.01) 

S = standard  deviation 

The  cumulative  probability  is  obtained  from  a table  containing  values  of  the 
standard  normal  distribution  for  various  Z-values. 

3.  Review  those  variables  to  which  the  indicated  decision  appears  sensitive 

and  determine  if  additional  information  should  be  obtained  to  improve  the 
confidence  of  the  assessor  that  the  decision  indicated  is  valid.  j 

4.  If  a "reject"  decision  is  indicated  and  affirmed  by  the  decision  maker, 
create  a tickler  file  entry  as  outlined  in  04.06,  otherwise  proceed  in 
the  analysis  to  Task  7. 
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SENSITIVITY  ANALYSIS 


VARIABLE 

MEAN  VALUE 

CONFIDENCE  RANGE 

BREAK-EVEN 

VALUE  FR0M/T01 

SENSITIVITY2 

PROBABILITY3 

Success  Probability  - U 

.7 

.5  - .8 

.54 

ACC -ACS 

32.7 

.05 

Success  Probability  - W 

. .8 

.65  - .95 

.94 

ACC -ACS 

12.0 

.01 

Importance  - Student 

Morale 

8 

★★★★★ 

6.7 

ACC-ACS 

5.6 

★*★★4 

Attrition  Rate 

(as  it  affects 

.02 

.00  - .07 

.06 

ACC-REJ 

1.3 

.03 

actual  dollars) 


1.  The  original  indicated  decision  is  shown  as  well  as  the  suggested  decision  beyond  the  break-even 
value. 

2.  Percent  change  in  the  decision  variable  (output)  for  each  percentage  change  in  the  input  variable. 

3.  The  probability,  based  upon  the  distribution  using  the  mean  and  confidence  limits,  that  the  decision 
will  change  as  a result  of  probable  variance  in  the  variable. 

4.  A distribution  could  not  be  established  because  of  the  absence  of  confidence  limits,  therefore, 
no  probability  was  generated. 
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FIGURE  6.7.1.  SAMPLE  SENSITIVITY  ANALYSIS  CHART 
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FIGURE  6.7.2.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  GRADUATE 
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FIGURE  6.7.3.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  STUDENT  POSITION 
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FIGURE  6.7.4.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  GRADUATE 
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FIGURE  6.7.5.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  STUDENT  POSITION 
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FIGURE  6.7.6.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  GRADUATE 
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FIGURE  6.7.7.  TRAINING  COST  MODEL  SENSITIVITY  ANALYSIS 
AVERAGE  DISCOUNTED  COST  PER  STUDENT  POSITION 
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TASK:  PERFORM  FINANCIAL  ANALYSIS 

Financial  analysis  adds  an  additional  dimension  to  the  data  available  for  the 
decision  maker  to  use  In  reaching  a conclusion  on  the  acceptance  or  rejection. 
The  cost-benefits  analysis  from  Task  6,  combined  with  the  following  financial 
analysis,  in  general,  meets  the  requirements  for  an  economic  analysis.  The 
definition  of  Economic  Analysis  as  stated  in  SECNAVINST  7000. 14A,  dated  14  Mar 
1973  (Enclosure  (1),  page  2),  is  as  follows: 

Economic  Analysis:  A systematic  approach  to  the  problem  of  choosing  how  to 

employ  scarce  resources  and  an  investigation  of  the  full  implications  of 
achieving  a given  objective  in  the  most  efficient  and  effective  manner.  The 
determination  of  efficiency  and  effectiveness  is  implicit  in  the  assessment 
of  the  cost-effectiveness  of  alternative  approaches  and  is  accomplished  by: 

a.  Systematically  identifying  the  benefits  and  other  outputs  and  costs 
associated  with  alternative  programs,  missions,  and  functions  and/or 
of  alternate  ways  for  accomplishing  a given  program  (usually  referred 
to  as  projects  and  activities). 

b.  Highlighting  the  sensitivity  of  a decision  to  the  values  of  the  key 
variables  and  assumptions  on  which  decisions  are  based  including 
technical,  operational,  schedule,  and  other  performance  considerations. 

t 

c.  Evaluating  alternative  methods  of  financing  investments,  such  as  lease 
or  buy;  and 

d.  Using  benefits  and  costs  to  compare  the  relative  merits  of  alterna- 
tives as  an  aid  in: 

1)  Making  trade-offs  between  alternatives, 

2)  Recommending  the  cost-effective  alternative,  and 

3)  In  establishing  or  changing  priorities. 

In  terms  of  this  definition,  the  entire  ETAM  can  be  considered  an  economic 
analysis.  This  task  focuses  on  the  quantifiable  benefits  (savings)  and 
liabilities  (investment  and  recurring  costs)  to  measure  the  justifiability  of 
the  proposed  innovation  in  terms  of  its  return  on  investment;  and  at  the  same 
time  it  provides  a comparative  analysis  of  several  alternatives  (identified  in 
Task  2).  The  “General  Guidelines  for  Doing  Economic  Analysis/Program  Evalua- 
tion Studies"  contained  in  SECNAVINST  7000. 14A  should  be  reviewed  In  conjunction 
with  this  task. 
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SUBTASK 

1.  Assemble  the  necessary  input  data  to  develop  the  required 
analyses. 

t 

2.  Develop  incremental  cost/savings  analysis  for  each 
alternative  procedure  to  produce  a Cost/Savings  Source 
Report. 

3.  Make  analysis  of  each  alternative  in  comparison  with  each 
other  alternative  to  produce  an  Alternatives  Analysis 
Report. 

4.  Perform  sensitivity  analysis  of  return  on  investment  to 
selected  variables. 

5.  Create  tickler  file  entry  identifying  criteria  for 
reassessment  of  innovation  if,  at  this  point,  the  proposal 
does  not  appear  feasible. 
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07.03.01 

07.04.01 
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TASK:  PERFORM  FINANCIAL  ANALYSIS 


SUBTASK:  ASSEMBLE  DATA 


DESCRIPTION 

The  assessor  may  or  may  not  have  carried  each  alternative  generated  in  Task  2 
through  the  entire  assessment  process.  If  each  alternative  has  been  subjected 
to  a rigorous  assessment,  then  the  cost  data  will  be  available  and  can  be 
introduced  directly  into  the  financial  analysis.  Otherwise  a detailed  cost 
analysis  will  have  to  be  performed.  The  data  from  the  proposed  innovation 
which  have  been  carried  through  the  assessment  process  to  this  point  will, 
however,  be  available. 

PROCEDURE 

1.  Obtain  the  cost  data  for  the  proposed  innovation  generated  by  the  cost 
model  run  in  06.03.  The  cost  data  should  reflect  the  results  of  Task  6 
which  should  have  led  to  selection  of  a particular  decision  tree  path. 

For  example,  if  the  particular  path  chosen  included  investment  in  addi- 
tional studies  or  R&D  projects,  these  costs  should  be  included  as  inputs 
to  the  cost  model . 

2.  Obtain  the  cost  data  for  the  baseline  system.  These  data  will  have  been 
used  in  Task  6 to  calculate  the  incremental  costs/savings  resulting  from 
the  proposed  innovation. 

3.  Obtain  data  available  for  any  alternatives  to  the  proposed  innovation 
which  may  have  been  generated  in  Task  2. 
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TASK:  PERFORM  FINANCIAL  ANALYSIS 

SUBTASK:  DEVELOP  COST/SAVINGS  ANALYSIS 

— 

DESCRIPTION 

In  this  sub task,  each  of  the  alternatives  will  be  subjected  to  an  incremental 
analysis  across  the  entire  range  of  application.  It  is  assumed  that  the 
alternative  approaches  were  designed  to  meet  the  same  objective  as  outlined 
in  Task  1 for  the  proposed  innovation;  therefore,  the  range-of-effect 
analysis  should  lead  to  the  same  results.  If,  however,  this  is  not  the  case 
then  a new  range-of-effect  analysis  must  be  performed  prior  to  costing  a 
particular  alternative. 

PROCEDURE 

1.  Determine  requirements  for  additional  cost  model  runs.  These  may  involve 
costing  of  alternatives  as  well  as  the  costing  of  the  specific  approach 
selected  for  the  proposed  innovation  as  a result  of  the  assessment 
process  in  Task  6. 

2.  Ensure  that  alternatives  have  been  subjected  to  the  appropriate  range-of- 
effect  analysis  so  that  the  entities  to  be  costed  (courses,  jobs,  equip- 
ments, etc.)  are  valid  targets  for  application  of  each  alternative 
proposal . 

3.  Establish  the  input  parameters  and  prepare  input  data  sheets  as  called 
out  in  06.03.  Each  alternative  will  be  incrementally  assessed  against 
every  other  alternative,  the  proposed  innovation,  and  the  baseline  (or 
existing)  system.  A model  run  input  is  prepared,  therefore,  for  each  of 
these  cases.  Two  input  factors  of  concern  at  this  point  are  the  infla- 
tion rate  and  the  discount  rate.  The  inflation  rate  permits  an  expression 
of  annual  figures  in  current  dollars;  i.e.,  the  dollar  values  that  would 
most  likely  be  reflected  in  the  budgets  for  those  outyears  where  prices 
have  increased.  The  discount  rate  permits  the  time  value  of  money  to  be 
considered  within  the  overall  decision  process.  The  Department  of 
Defense  and  Secretary  of  tne  Navy  have  issued  guidance  on  the  appropriate 
discount  rate  to  be  used.  The  present  rate  stated  in  D0D  INSTRUCTION 
7041.3  and  in  SECNAV  INSTRUCTION  7000. 14A  is  10%.  This  rate  can  be 
modified  if  the  analyst  feels  another  rate  is  more  appropriate.  In  the 
private  sector,  this  rate  is  sometimes  adjusted  to  reflect  risk.  For 
example,  low  risk  investments  such  as  in  replacement  of  machinery  may  use 
a discount  rate  as  low  as  the  current  going- rate  of  interest.  High  risk 
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investments  such  as  for  R&D  or  new  market-entry  products  may  use  a dis- 
count rate  that  is  much  higher;  i.e.,  25%  or  more.  If  an  alternate  rate 
is  selected,  model  runs  should  be  made  at  both  the  alternate  rate  and  the 
current  DOD  approved  rate;  i.e.,  10%. 

4.  Complete  model  runs  for  each  of  the  cases  prepared  in  step  3,  and  generate 
a Cost/Savings  Source  Report.  A sample  of  this  report  is  shown  in 
Figure  7.2.1. 

COMMENT:  The  Cost/Savings  Source  Report  shown  in  Figure  7.2.1  is  one  of  three 

formats  which  will  be  discussed  in  Section  IV  of  this  report  under  Computer 
Applications. 

The  values  of  costs  and  savings  represent  the  incremental  change  or 
difference  in  the  comparison  of  the  effects  of  one  alternative  against 
the  effects  of  another  alternative. 

5.  Review  the  total  and  present  value  (NONDISCOUNTED  TOTAL  $ and  PRESENT 
VALUE-CONST  $)  for  each  comparative  incremental  analysis.  If  these 
figures  are  not  positive,  then  the  incremental  rate  of  return  for  the 
alternative  in  question  will  not  be  to  the  level  expected;  therefore,  it 
may  be  desirable  to  exclude  this  alternative  from  further  analysis.  The 
information  regarding  any  alternatives  rejected  at  this  point  should,  how- 
ever, be  carried  forward  for  the  review  of  the  decision  maker  in  Task  8. 


COST/SAVINGS  SOURCE  REPORT 


INPUT: 

DISPLAY  FORMAT  - COURSE/EQUIPMENT/ JOB 
PLANNING  PERIOD  - 05  YEARS 


OUTPUT: 


CST(-)/SAV(+)  SOURCE  - 

TOTAL 

YEAR 

COURSES 

536B 

67.8 

7. 

536C 

61.2 

7. 

536D 

102.5 

8. 

JOBS 

EN 

52.4- 

11. 

EM 

51.8- 

12. 

MM 

89.3- 

24. 

NONDISCOUNTED  TOTAL  $ 

38.0 

24. 

PRESENT  VALUE-CONST  $ 

18.9 

21. 

INFLATED  TOTAL  $ 

54.1 

26. 

PRESENT  VALUE-CURNT  $ 

28.1 

23. 

DISCOUNT 

RATE  - 15  PERCENT 

INFLATION 

RATE  - 08  PERCENT 

YEAR  2 

YEAR  3 

YEAR  4 

YEAR  5 

21.4 

18.6 

13.2 

7.0 

22.3 

13.9 

12.8 

5.1 

30.2 

24.0 

22.2 

17.3 

16.6- 

14.1- 

6.1- 

4.1- 

18.2- 

10.6- 

7.4- 

3.0- 

17.1- 

18.0 

17.6- 

12.6- 

22.0 

13.8 

17.1 

9.7 

16.6 

9.1 

9.8 

4.8 

25.7 

17.4 

23.3 

14.3 

19.4 

11.4 

13.3 

7.1 

1 

6 

1 

8 

5- 

6- 

0- 

6- 

4 

6- 

1- 


FIGURE  7.2.1  COST/SAVINGS  SOURCE  REPORT 
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TASK:  PERFORM  FINANCIAL  ANALYSIS 


SUBTASK:  DEVELOP  ALTERNATIVES  COMPARISON  ANALYSIS 


DESCRIPTION 

The  purpose  of  this  subtask  is  to  organize  the  data  from  Subtask  07.02  in  such 
a way  that  alternatives  (including  the  proposed  innovation)  which  have  been 
deemed  reasonable  can  be  compared  with  one  another.  The  decision  maker  will 
have  available  certain  financial  measurements  which  are  indications  of  the 
relative  value  of  the  alternatives  under  consideration. 

PROCEDURE 

1.  Select  the  analytic  results  from  Subtask  07.02  for  those  alternatives 
accepted  for  further  consideration. 

2.  Organize  the  data  so  that  the  incremental  annual  costs  and  savings  are 
separated  into  the  following  categories: 

• R&D  costs  - while  these  costs  are  not  specifically  displayed  as  an 
output  of  this  subtask,  they  should  be  separately  identified  for 
subsequent  funding  requests  should  the  proposal  be  accepted. 

• Investment  costs  - these  will  be  primarily  equipment  purchase  costs; 
however,  other  costs  may  be  considered  investment,  such  as  start-up 
costs  or  rehabilitation  and  modification  costs.  The  assessor  should 
refer  to  SECNAV  INSTRUCTION  7000. 14A  for  a more  complete  identifica- 
tion of  the  types  of  costs  which  might  fall  into  this  category. 

• Recurring  costs  - these  are  costs  expected  to  recur  on  an  annual  basis, 
such  as  costs  for  personnel,  maintenance  of  equipment,  maintenance  of 
instructional  materials,  supplies,  etc. 

COMMENT:  Data  should  have  been  classified  according  to  the  preceding  cate- 

gories when  originally  introduced  into  the  costing  cycle  in  Subtasks  06.02 
and  06.03.  No  major  reorganization  should  be  required  at  this  time. 

3.  Generate  an  Alternatives  Analysis  Report  as  presented  in  Figure  7.3.1. 

This  report  sunmarizes  the  incremental  investment  and  recurring  costs,  and 
the  savings  for  up  to  three  alternatives,  one  of  which  may  be  the  baseline 
or  existing  system.  The  financial  measurements  calculated  and  displayed 
as  a part  of  the  Alternatives  Analysis  Report  are: 

• Costs  and  savings  are  presented  both  in  nondiscounted  constant  dollars 
and  their  present  value,  as  obtained  by  discounting  at  the  discount 
rate  (percent)  indicated. 
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• A savings  investment  ratio  is  calculated  as  the  ratio  of  the  PV  net 
recurring  cst(-)/sav(+)  to  the  total  PV  of  new  investment(-) . 

• The  rate  of  return  on  investment  is  the  rate  which,  if  substituted 
for  the  discount  rate  used,  would  have  made  the  total  PV  of  cst(-)/ 
sav(+)  equal  to  zero;  i.e.,  the  present  value  of  the  savings  would 
have  equaled  the  present  value  of  the  investment. 

• The  uniform  annual  cst(-)/sav(+)  is  the  average  cost  of  savings  spread 
over  the  economic  life  of  the  alternative  used  for  the  analysis.  It 
becomes  a useful  number  for  comparing  alternatives  which  have  varying 
economic  lives. 

COMMENT:  The  computational  aspects  of  this  report  and  its  relationship  to  the 

Cost/Savings  Source  Report  will  be  discussed  in  further  detail  in  Section  IV 
of  this  report  under  Computer  Applications. 

4.  Review  the  alternatives  being  compared  for  those  having  favorable  return 
on  investment.  If  the  proposed  innovation  shows  the  highest  rate  of 
return,  then  there  should  be  sufficient  basis  for  its  acceptance.  How- 
ever, the  following  should  be  considered  where  the  proposed  innovation 
does  not  show  the  highest  rate  of  return  when  compared  with  either  the 
baseline  (or  existing)  system  or  with  one  of  the  alternatives. 

• The  amount  of  total  investment  required  to  accomplish  each  of  the 
alternatives  may  be  a critical  factor  in  the  ultimate  decision  to 
accept  one  or  the  other.  In  Tasks  3 and  4,  the  funding  as  a possible 
constraint  was  assessed  in  terms  of  the  potential  risk  it  imposed  on 
the  implementation  and  use  of  the  proposed  innovation.  An  alternative, 
even  with  a lower  rate  of  return,  may  be  a more  favorable  approach  if 
the  overall  risk  is  significantly  lower  due  to  a lesser  funding 

requi rement. 

• One  concern  in  financial  analysis  is  with  the  rate  of  return  obtained 
on  the  investment  of  incremental  dollars.  If  the  rate  of  return  on 
the  incremental  investment,  in  going  from  one  alternative  to  another, 
is  less  than  could  be  obtained  through  some  alternate  form  of  invest- 
ment, then  perhaps  the  additional  expenditure  should  be  rejected.  If, 
however,  the  investment  dollars  could  not  be  put  to  better  use,  perhaps 
the  proposal  with  the  higher  investment  requirements  should  be  accepted, 
even  though  its  rate  of  return  is  less  than  an  alternative  with  a 
lesser  investment  requirement. 

• If  an  alternative  proposal  looks  more  favorable  on  a financial  basis, 
it  should  be  subjected  to  the  analytic  steps  outlined  in  Task  6 to 
determine  if  it  might  be  rejected  because  of  greater  nonquantified 
liabilities  or  because  there  is  a change  in  the  probabilities  that 

the  alternative  can  be  successfully  implemented,  or  successfully  applied, 
by  the  user. 
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5.  Determine  the  alternative  or  alternatives  to  be  considered  in  the  ultimate 
decision  process  and  perform  the  required  sensitivity  analysis  outlined  in 
Subtask  07.04. 

6.  If  no  candidates  have  an  acceptable  level  of  financial  return  and  there  is 
no  hard  requirement  for  selection  of  one  of  them  for  "mission  accomplish- 
ment" reasons,  then  a tickler  file  entry  should  be  created  as  outlined  in 
Subtask  04.06. 
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TASK:  PERFORM  FINANCIAL  ANALYSIS 


SUBTASK:  PERFORM  SENSITIVITY  ANALYSIS 


DESCRIPTION 

The  results  of  comparing  alternatives  may  have  shown  several  to  be  fairly 
close  to  each  other  in  terms  of  their  rate  of  return,  or  the  selected  candidate 
may  have  had  a rate  of  return  close  to  the  required  return  as  stated  in  the 
discount  rate.  In  either  of  these  cases,  the  assessor  may  want  to  know  which 
variables  are  particularly  sensitive,  such  that  their  variation  may  possibly 
change  the  ultimate  decision.  The  purpose  of  this  subtask  (as  with  the  pre- 
ceding sensitivity  analysis  in  Subtask  06.07)  is  to  provide  the  assessor  with 
insight  into  the  variables  which  could  cause  a change  in  the  decision  if  they 
were  to  vary  over  a reasonably  expected  range.  A need  for  improving  the  con- 
fidence in  the  estimate  of  a particular  variable  may  be  indicated  by  this 
analysis  which  in  turn  may  suggest  the  gathering  of  additional  information. 

PROCEDURE 

1.  Review  the  sensitivity  analysis  diagrams  for  the  models  used  to  develop 
the  cost  factors  to  determine  if  any  of  the  variables  affected  by  the 
proposed  innovation  are  particularly  sensitive.  Sample  sensitivity 
graphs  of  the  Training  Cost  Model  were  presented  in  Subtask  06.07. 

2.  Identify  the  variables  to  be  inputted  to  the  sensitivity  analysis  program 
which  will  perform  the  following  steps. 

a.  Determine  if  the  inputted  variables  cause  a change  in  the  decision 
variable  beyond  some  expressed  limit,  first  considering  the  limits 
of  the  confidence  range  expressed  for  the  variables.  A confidence 
range  should  have  been  expressed  for  each  of  the  variables  considered 
relevant  to  the  application  of  the  proposed  innovation.  However,  in 
the  absence  of  a confidence  range,  the  variable  value  at  which  the 
limit  of  the  decision  variable  is  reached  will  be  calculated. 

b.  The  break-even  value,  or  point  at  which  the  decision  variable  limit 
is  reached,  will  be  calculated  for  each  variable  in  which  this  occurs 
in  between  the  two  confidence  limits. 

c.  A sensitivity  and  probability  that  the  indicated  decision  will  change, 
will  be  calculated  for  each  variable  that  causes  a change  within  the 
confidence  range. 

% 

d.  A report  similar  to  that  shown  in  Subtask  06.07,  Figure  6.7.1,  will  be 
generated  for  the  assessor  to  review. 
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COMMENT:  The  cautions  in  interpretation  of  the  statistical  data  mentioned  in 

Subtask  06.07  apply  here,  also,  and  they  should  be  reviewed. 

5.  Review  those  variables  to  which  the  indicated  decision  appears  sensitive 
and  determine  if  additional  information  should  be  obtained  to  improve  the 
confidence  of  the  assessor  that  the  decision  indicated  is  valid. 

6.  The  information  reports  generated  in  this  task  will  be  used  in  Task  8 as 
a part  of  the  overall  decision-making  format.  If,  at  this  time,  however, 
there  appears  to  be  an  obvious  basis  for  rejecting  the  proposed  innovation 
or  any  of  the  alternatives,  a tickler  file  entry  as  outlined  in  Subtask 
04.06  should  be  created. 
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SUMMARY  OF  RELEVANT  FACTORS  CONSIDERED 
IN  THE  ECONOMIC  ANALYSES: 


Comparative  Economic  Life  of 
Alternatives  (Years) 

Total  Nondisc  Net  Investment 
Total  Nondisc  Recurring  Cst/Sav 

Present  Value  of  New  Investment 
Plus:  PV  Existing  Assets  Used 

Less:  PV  Existing  Assets  Repl 

Less:  PV  Investment  End  Value 

Total  PV  of  New  Investment(-) 

PV  Net  Recurring  Cst(-)/Sav(+) 
Plus:  PV  Refurb/Modif  Saved 

Total  PV  of  Cst(-)/Sav(+) 

Discount  Rate  (Percent) 

Savings  Investment  Ratio 
Rate  of  Return  on  Investment 
Uniform  Annual  Cst(-)/Sav(+) 


EXISTING  VS.  EXISTING  VS.  PROPOSED  VS. 


PROPOSED 

ALTERNATIVE 

ALTERNATIVE 

5 V 

5 V 

5 V 

S 5 

S 5 

S 5 

100.0- 

40.0- 

60.0- 

220.0 

100.0 

120.0 

100.0- 

40.0- 

60.0- 

0.0- 

4.3- 

0.0 

4.3 

0.0 

0.0 

14.9 

2.5 

12.4 

80.8- 

41.8- 

47.6- 

143.4 

68.0 

75.3 

0.0 

0.0 

0.0 

62.6 

26.2 

* 27.7 

15 

15 

15 

1.77 

1.63 

1.58 

35.9 

38.4 

29.2 

18.7 

7.8 

8.3 

FIGURE  7.3.1.  ALTERNATIVE  ANALYSIS  REPORT 
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TASK:  MAKE  ACCEPT/REJECT/STUDY  DECISION 

This  task  is  to  provide  guidance  to  the  assessor  in  order  that  a coherent 
decision-making  format  can  be  structured  for  presentation  to  the  ultimate 
decision  makers.  It  provides  examples  of  both  format  and  content  which  are 
recommended  for  presentation.  In  addition,  it  addresses  some  of  the  questions 
the  decision  maker  might  have,  in  order  to  suggest  back  up  detail  that  should 
also  be  made  available  as  a supplement  to  the  presentation.  The  objective  of 
ETAM  has  been  to  ensure  that  information  has  not  been  lost  in  the  process  of 
quantifying  and  summarizing  the  data.  Therefore,  the  information  available 
to  the  decision  maker  will  be  in  consonance  with  this  objective. 

SUBTASKS  REFERENCE 


1.  Assemble  the  outputs  from  each  step  of  the  assessment  08.01.01 

procedure,  Tasks  1 through  7. 

2.  Define  the  presentation  format  so  that  it  is  consistent  08.02.01 
with  the  requirements  of  the  decision  maker. 

3.  Prepare  the  material  to  be  presented  to  the  decision  08.03.01 

maker. 


4.  Review  the  material  planned  for  presentation  in  the  08.04.01 

context  of  potential  questions  which  the  decision 

maker  might  have. 

5.  Obtain  the  decision  and  define  a plan  for  proceeding  08.05.01 

based  upon  the  decision. 
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TASK:  MAKE  ACCEPT/REJECT/STUOY  DECISION 


SUBTASK:  ASSEMBLE  TASK  OUTPUT  DATA 


DESCRIPTION 

Assemble  the  descriptive  and  analytic  data  resulting  from  each  of  the  ETAM 
tasks.  These  data  should  have  been  maintained  in  a project  file  and  should, 
therefore,  be  readily  available. 

PROCEDURE 

1.  Obtain  the  following  data  which  will  be  used  in  preparing  the  presentation 

to  the  decision  maker. 

a.  Task  1 - all  information  describing  the  proposed  innovation  and  any 

of  the  alternatives  subjected  to  the  procedures  in  Task  1.  Of  primary 
interest  are  the  objectives  of  the  innovation  and  any  assumptions  made 
with  regard  to  the  application  of  the  innovation. 

b.  Task  2 - any  alternatives  which  were  considered,  whether  they  were 
subjected  to  the  entire  assessment  procedure  or  not. 

c.  Task  3 - the  results  of  the  preliminary  risk  assessment  identifying 
major  stumbling  blocks  to  the  successful  application  of  the  innovation. 

d.  Task  4 - the  specific  projects  defined  in  order  to  reduce  identified 
risks,  and  the  assessment  of  costs  and  impact  upon  risk  reduction. 

e.  Task  5 - the  search  parameters  derived  from  evaluating  the  innovation 
in  relation  to  a number  of  taxonomic  structures,  the  particular 
entities  found  as  a result  of  the  searches,  and  the  benefits  pattern 
attached  to  the  successful  application  of  the  proposed  innovation. 

f.  Task  6 - the  results  of  each  analytic  step  which  led  to  the  formulation 
of  the  final  decision  tree  structure,  and  the  sensitivity  analysis  data 
related  to  the  decision  tree. 

g.  Task  7 - the  Cost/Savings  Source  Report  for  each  of  the  alternatives 
under  consideration,  and  the  Alternatives  Analysis  Report  comparing 
alternatives  on  a financial  basis.  Sensitivity  analysis  data  relative 
to  the  selected  alternatives  should  also  be  obtained. 

h.  Any  additional  references,  supporting  rationale,  or  expert  judgments 
obtained  in  the  assessment  process  should  be  obtained. 
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TASK:  MAKE  ACCEPT/REJECT/STUDY  DECISION 


SUBTASK:  DEFINE  PRESENTATION  FORMAT 


DESCRIPTION 

The  decision  maker  will  undoubtedly  have  a limited  amount  of  time  to  review 
the  proposal  and  the  analytic  results.  Also,  the  decision  maker  will  have 
format  preferences  which  aid  in  the  comparison  of  numerous  requests  for 
resources.  A third  consideration  is  the  level  of  technical  understanding  of 
the  problem  and  of  concepts  inherent  in  the  solution  that  the  decision  maker 
might  have.  The  purpose  of  this  subtask  is  to  provide  guidance  to  the 
assessor  in  arriving  at  a presentation  format  which  meets  the  above  needs. 

PROCEDURE 

1.  Obtain  information  on  the  preferences  of  the  decision  maker  for  particular 
types  of  formats.  These  preferences  should  be  in  terms  of  medium  and 
style. 

2.  Determine  the  amount  of  time  which  will  be  available  for  the  review  and 
decision  process  and  the  conditions  under  which  the  review  will  be 
conducted. 

3.  Determine  the  degree  of  background  which  the  decision  maker  has  in  both 
the  problem  addressed  by  the  innovation,  and  in  the  various  technical 
aspects  of  the  solution. 

4.  Define  a format  which  considers  the  needs  of  the  decision  maker,  but  at 
the  same  time  ensures  that  the  relative  merits  of  the  proposal  can  be 
adequately  displayed.  Some  presentation  techniques  are  outlined  in  a 
Department  of  Defense  publication  titled,  "Economic  Analysis  Handbook, 

2nd  Edition." 

RATIONALE 

This  final  step  of  presenting  the  results  of  substantial  analytic  work  may 
have  a significant  effect  upon  the  decision  outcome.  Even  a sound  and  justi- 
fied proposal  may  fail  to  gain  acceptance  because  of  shortcomings  in  the 
presentation  format. 
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TASK:  MAKE  ACCEPT/REJECT/STUDY  DECISION 

SUBTASK:  PREPARE  MATERIAL  FOR  PRESENTATION 


DESCRIPTION 

Once  a format  has  been  defined  for  the  presentation,  the  content  of  the 
analysis  may  have  to  be  restructured  to  fit.  It  is  important,  though,  that 
the  final  content  be  organized  so  that  the  decision  maker  is  presented  with 
a coherent  view  of  the  innovation,  its  objective  in  terms  of  application,  the 
systematic  analysis  to  which  it  was  subjected,  the  role  of  experts  in  making 
critical  aspects  of  the  analysis,  and  of  the  final  conclusions  and  recommenda- 
tions resulting  from  the  analysis.  The  purpose  of  this  subtask  is  to  provide 
the  assessor  with  guidance  in  structuring  the  content  for  presentation  within 
the  identified  format. 

PROCEDURE 

1.  Organize  the  presentation  sequence  so  that  it  represents  a logical  pro- 
gression from  a statement  of  the  problem,  in  terms  that  the  decision 
maker  will  understand,  to  any  background  data  necessary  for  understanding 
why  the  objective  of  the  innovation  is  reasonable,  to  a statement  of  the 
objective,  to  the  proposed  approach  which  meets  the  stated  objective,  and 
finally  to  the  various  supporting  data  necessary  for  the  decision. 

2.  Prepare  conclusions  which  are  consistent  with  the  supporting  analytic  data 
and  recomendations  which  support  the  conclusions. 

3.  Consider  alternative  levels  of  presentation  which  will  still  retain  the 
coherence  intended. 

4.  Prepare  a concise  summary  which  reiterates  the  general  conclusions  and 
recommendations  and  includes  a basic  supporting  rationale  for  each. 
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TASK:  MAKE  ACCEPT/REJECT/STUDY  DECISION 


SUBTASK:  REVIEW  PRESENTATION  MATERIAL 


DESCRIPTION 

The  purpose  of  this  subtask  is  to  suggest  possible  questions  that  the  decision 
maker  might  have  with  regard  to  the  application  of  the  innovation,  the  analytic 
process  followed  and  the  data  inputs  to  it,  and  conclusions  and  recommendations. 
It  is  assumed  that  the  assessment  process  has  considered  all  of  these  potential 
concerns  of  the  decision  maker  at  one  stage  of  the  procedure  or  another.  There- 
fore, the  intent  here  is  not  so  much  to  ensure  that  they  have  been  adequately 
addressed,  but  rather  to  serve  as  a guide  to  the  presenter  in  preparing  for 
the  presentation. 

PROCEDURE 

1.  Consider  the  following  questions  in  reviewing  the  material  to  be  presented 
to  the  decision  maker.* 

a.  Objective,  assumptions,  and  alternatives. 

Is  the  problem  stated  the  real  problem? 

Are  all  reasonable  assumptions  identified  and  explained? 

Are  assumptions  too  restrictive?  Too  broad? 

Are  intuitive  judgments  identified  as  such?  Are  uncertainties  treated 
as  facts?  Can  the  facts  be  verified? 

Are  any  feasible  alternatives  omitted? 

Are  the  alternatives  well-defined  and  discrete?  Do  they  overlap? 

b.  Benefit/liability  determination. 

Were  all  relevant  benefits  and  liabilities  identified? 

Was  the  appropriate  level  of  relative  importance  assigned  to  each 
benefit/liability? 

Were  the  criteria  used  to  measure  benefit/liability  justified  by  the 
context  of  the  study? 

Was  the  benefit/liability,  in  fact,  unmeasurable?  Has  there  been  a 
rational  assessment  of  nonquantifiable  factors? 

Was  expert  opinion  used?  Were  these  experts  properly  qualified? 


♦Adapted  from  the  "Economic  Analysis  Handbook,  2nd  Edition,"  prepared  by  the 
Department  of  Defense. 
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c.  Risk  assessment. 

Were  all  potential  risk  factors  considered? 

Was  the  appropriate  level  of  risk  attached  to  each? 

Are  R&D  or  study  projects  to  address  risk  factors  reasonable? 

Was  the  reassessment  of  risk  given  these  projects  reasonable? 

d.  Cost  estimates. 

What  costing  method  was  used?  It  is  appropriate? 

Are  all  relevant  costs  included?  Are  overhead  costs  appropriately 
included  or  excluded? 

Is  the  rationale  for  including  or  excluding  certain  costs  included? 
Are  the  opportunity  costs  properly  identified?  Are  sunk  costs 
excluded? 

Are  the  sources  of  cost  data  included?  Are  they  accurate? 

Are  the  cost  estimating  relationships  valid,  if  the  parametric  method 
was  used?  Are  extrapolations  used  without  proof? 

Was  an  appropriate  discount  rate  used  for  obtaining  present  value? 

2.  Prepare  any  supplementary  material  suggested  by  the  potential  questions 
which  the  decision  maker  might  have. 
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TASK:  MAKE  ACCEPT/REJECT/STUDY  DECISION 


SUBTASK:  OBTAIN  DECISION  AND  DEFINE  PLAN 


DESCRIPTION 

The  preceding  steps  should  lead  to  either  a firm  or  a tentative  decision  upon 
which  some  degree  of  follow-on  activity  can  be  based.  This  subtask  suggests 
some  of  the  follow-on  activities  to  be  pursued  based  upon  different  decision 
outcomes . 

PROCEDURE 

1.  Obtain  a decision  (firm  or  tentative)  to  accept,  continue  to  study,  or 
reject  the  proposed  innovation  or  an  alternative. 

2.  Perform  the  following  activities  based  upon  the  decision  outcome. 

a.  Decision  to  accept. 

Prepare  a detailed  implementation  with  supporting  rationale  for  each 
phase  of  development,  implementation,  and  testing. 

Identify  resource  needs. 

Identify  checkpoints  and  major  milestones. 

Develop  auditing  procedures. 

b.  Decision  to  accept  with  continued  study. 

This  decision  requires  essentially  the  same  items  mentioned  above, 
except  that  R&D  and  study  plans  may  have  to  be  developed,  and  go  no-go 
criteria  established  if  appropriate  to  the  intent  of  the  R&D  or  study 
projects . 

c.  Decision  to  reject. 

Document  rationale  for  rejection. 

Estimate  potential  for  and  timing  of  future  application. 

Create  tickler  file  entry  per  Subtask  04.06. 
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SECTION  IV 

COMPUTER  APPLICATIONS 


The  Educational  Technology  Assessment  Model  Is  an  organized  grouping  of  proce- 
dural sequences  that  provide  a structure  for  efficiently  and  effectively 
assessing  the  true  value  of  a proposed  innovation  In  training.  Efficiency  Is, 
however,  highly  dependent  upon  the  assessor's  ability  to  access  data  when 
required,  rapidly  make  calculations  using  selectable  routines  (e.g..  Training 
Cost  Model),  modify  input  data  and  make  recalculations,  and  obtain  output 
reports  with  suitable  content  and  format  for  effective  analysis  and  decision 
making.  These  requirements  for  an  efficient  assessment  processing  structure 
suggest  the  need  for: 

1.  Highly  organized  data  bases  with  rapid  search  and  retrieval  capability. 

2.  A collection  of  automated  routines  which  can  be  selected  at  specific 
stages  of  the  assessment  process  for  making  calculations  or  manipulating 
data. 

3.  A project  maintenance  facility  which  will  serve  as  a repository  for: 

a.  Descriptive  data  on  the  Innovation. 

b.  Input  data  factors. 

c.  Results  of  data  base  searches. 

d.  Intermediate  calculated  results. 

e.  Implementation  subproject  descriptive  and  cost  data. 

f.  Final  analytic  results. 

4.  An  "Executive  Control  System"  capable  of  sequencing  data  retrieval,  data 
manipulation,  calculations.  Interactive  requirements,  and  output  genera- 
tion based  upon  the  analytic  needs  at  a particular  phase  of  the  assess- 
ment. 

The  purpose  of  this  section  Is  to  describe  the  general  framework  of  an  auto- 
mated data  system  capable  of  fulfilling  these  needs,  and  to  provide  develop- 
ment guidance  In  the  form  of  design  specifications  for  those  who  might  under- 
take the  modeling  and  programming  of  such  a system. 

ETAM  AUTOMATED  DATA  SYSTEM  STRUCTURE 

The  automated  data  system  Is  composed  of  five  major  programs  which  control  the 
flow  of  data  and  sequence  of  calculations,  as  well  as  provide  Input  data  paths 
to  the  data  bases.  The  Interrelationships  between  these  five  programs  and  the 
data  bases  are  shown  In  Figure  IV-1.  Following  are  brief  descriptions  of  each 
of  the  programs. 
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DESCRIPTION  STRUCTURING  PROGRAM.  Source  descriptive  and  project  data  are 
entered  via  the  Description  Structuring  Program  which  performs  a missing  data 
edit  and  identifies  either  the  need  for  additional  data  or  the  default  data 
pattern.  Project  Identifications  are  established  to  ensure  traceability 
throughout  the  entire  analytic  and  data  transfer  process.  Overall  data  struc- 
turing, as  it  relates  .to  the  organization  of  the  project  file  data  base,  is 
performed  here.  Any  batch  data  input  (e.g.,  cost  model  input  parameters)  is 
entered  through  the  Description  Structuring  Program. 

RANGE-OF-EFFECT  PROGRAM.  The  purpose  of  this  program  is  to  formulate  the  pro- 
cessing pattern  which  is  Inputted  to  the  Assessment  Program.  It  will  provide 
the  user  with  preliminary  guidance  in  preparing  the  taxonomic  descriptors 
which  will  be  used  to  search  out  the  innovation’s  full  range  of  application. 

Once  the  search  characteristics  have  been  defined,  either  interactively  or  off- 
line, the  specific  search  parameter  for  the  equipment,  course,  or  job  data 
bases  are  generated.  The  Range-of-Effect  Program  also  structures  and  edits  the 
benefits  pattern  to  be  assigned  specific  attributes  of  entitles  selected  from 
the  data  bases;  e.g.,  magnitude,  importance,  and  confidence  in  the  attrition 
factor  for  course  12AB.  Specific  allocation  rules,  which  provide  further 
selectivity  to  the  final  Assessment  Program  input,  are  formulated;  e.g., 
applies  only  to  attritions  Identified  with  a prior  setback.  Some  Innovation 
benefits  will  be  direcly  costable,  which  may  preclude  the  need  for  allocation 
rules  or  benefits  patterns,  other  than  the  direct  amount  of  dollars  displaced. 
The  project  file  data  base,  containing  abbreviated  records  for  all  Navy  courses, 
jobs,  etc.,  and  user  interactive  inputs  are  the  primary  sources  of  data  upon 
which  the  Range-of-Effect  Program  operates.  Any  of  the  entities  extracted 
through  a data  search  operation  will  be  passed  to  the  extracted  data  base  for 
further  manipulation. 

ASSESSMENT  PROGRAM.  The  Assessment  Program  performs  major  computational  steps 
in  the  analytic  process.  It  contains  four  basic  types  of  computational  capa- 
bilities, namely,  cost  models,  decision  tree  evaluators,  financial  analysis  rou- 
tines, and  sensitivity  analysis  routines.  In  conjunction  with  these  computa- 
tional routines,  basic  Input  editing  and  default  data  selection  functions  are 
carried  out  within  the  Assessment  Program.  These  routines  will  communicate 
with  the  user  to  Identify  additional  data  requirements,  or  to  point  out  the 
characteristics  of  particular  default  data  being  used.  Interim,  as  well  as 
final  results,  will  be  retained  in  the  ETAM  Working  Data  Base  for  additional 
processing  or  modification.  If  desired.  An  aggregation  routine  will  provide 
the  means  of  summarizing  the  data  from  multiple  selected  entities  (e.g.,  jobs), 
so  that  the  computational  routines  can  be  run  on  aggregated  data. 

USER  INTERACTIVE  PROGRAM.  This  program  is  effectively  the  "Executive  Control 
System"  for  the  entire  system.  It  includes  a skeleton  monitor  program  which 
controls  the  sequence  of  operation  for  the  various  routines  identified  in  the 
other  major  programs.  The  User  Interactive  Program  also  contains  input-output 
routines  for  communicating  with  the  user  through  a terminal  and  graphic  display, 
or  through  a set  of  standard,  selectable  reports,  produced  as  output  from  the 
various  routines  in  the  major  programs.  In  certain  routines,  such  as  the  deci- 
sion tree  evaluation,  within  the  assessment  program,  interaction  with  the  user 
may  be  highly  desirable  in  formulating  parameters  or  In  generating  the  scenarios 
required  for  each  decision  tree  outcome.  The  routines  to  accomplish  this  Inter- 
action are  a part  of  the  User  Interactive  Program. 
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DATA  BASE  MAINTENANCE  PROGRAM.  This  Is  a program  containing  a number  of  stan- 
dard routines  required  to  maintain  the  Master  Data  Bases.  They  provide  the 
facilities  for  initial  file  loading,  file  updating,  file  editing,  and  for 
unloading  or  dumping  the  file  to  some  output  device.  A similar  set  of  main- 
tenance routines  will  also  be  required  for  the  ETAM  Working  Data  Base. 

DATA  BASE  CONSIDERATIONS 

The  general  detail  relating  to  the  structure  of  the  ETAM  data  bases  Is  covered 
In  the  portion  of  this  section  titled  "ETAM  PROGRAM  DESIGN  SPECIFICATION."  The 
discussion  here  Is  limited  to  data  base  considerations  that  apply  only  to  the 
external  use  of  information  In  the  training  enterprise  for  retrieval  and  ana- 
lytic purposes.  That  Is,  the  major  concern  here  Is  with  the  disciplines  for 
the  naming  of  classes  of  data  as  users  will  Interact  with  those  names  and  the 
data.  The  Internal  logic  and  coding  of  data  structures  will  not  be  discussed. 

PURPOSE.  The  specific  purposes  for  examining  data  base  factors  here  are 
directed  towards  the  Issues  of  matching  the  characterization  of  a proposed 
innovation  or  change  In  training  practice  or  technology,  and  the  characteriza- 
tions of  what  are  or  may  be  affected  by  that  potential  change.  Thus,  a nomen- 
clature Is  required  for  the  classification  of  the  various  kinds  and  patterns  of 
benefits  and  liabilities  that  are  associated  with  any  aspect  of  the  instruction- 
al process.  If  the  nomenclature  is  to  serve  additional  practical  purposes  for 
evaluation,  the  variables  and  entities  that  are  Identified  should  be  associated 
with  cost  factors.  Thus,  a given  benefit  factor  can  be  directly  associated 
with  cost  because  it  is  also  a cost  factor,  and  a term  in  the  classification 
scheme  can  be  used  both  to  access  data  about  benefits  accruing  from  that  term, 
and  also  data  about  the  costs  associated  with  that  term.  The  value  of  this 
capability  in  assessing  the  desirability  of  a proposed  change  In  method  or 
resource  used  should  be  clear. 

But  the  classification  scheme  should  not  only  Identify  Individual  benefits  and 
patterns  of  benefits,  and  costs  associated  with  those  benefits  for  the  purpose 
of  a technique  for  evaluating  proposed  changes  in  Instructional  technology.  It 
should  also  represent  the  conceptual  structure,  or  model,  of  Instructional 
technology  In  Its  broadest  sense.  In  this  sense,  the  data  base  classification 
scheme  becomes  a taxonomy. 

The  treatment  here  Is  not  a rationale  for  a taxonomic  structure  In  educational 
technology,  nor  will  It  contain  justification  for  the  nomenclature  proposed 
here.  It  will  not  represent  all  of  the  cost  variables  of  the  training  enter- 
prise by  any  means  --  rather  It  will  Identify  the  major  functional  variables 
In  education  and  training  around  which,  for  some  purposes,  cost  data  should  be 
clustered.  These  purposes  are  clustered  In  the  potential  manifestations  of 
changes  In  educational  technology. 

Specifically,  the  purpose  of  the  nomenclatures  to  be  proposed  In  the  following 
paragraphs  is  to  provide  key  terms  to  apply  to  the  description  of  Instructional 
vehicles  and  media,  to  students  and  job  Incumbents,  to  jobs  and  job  tasks,  and 
to  courses  and  constituent  course  units  and  objectives.  It  Is  not  proposed 
that  only  those  descriptors  or  Index  terms  proposed  will  be  sufficient  for  all 
purposes  In  the  use  of  the  training  system's  data  bases. 
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The  Data  Sets.  A data  set  may  be  thought  of  as  a tabular  arrangement  of  data 
In  which  a primary  list  of  entities,  such  as  student  name,  is  associated  with 
a set  of  attribute  names,  such  as  "date  of  enlistment,"  "training  courses  taken," 
"job  skills,"  and  so  forth.  An  entry  in  the  table  under  an  attribute  may  be 
quantitative,  qualitative,  or  text  description.  The  key  factor  In  the  practical 
use  of  a data  set  is  in  the  name  of  the  primary  entities  and  the  names  of  the 
attributes  that  make  up  the  headings  in  the  table.  The  name  of  a data  set  is 
generally  based  on  the  name  of  the  primary  entity. 

The  benefits  and  liabilities  analysis  In  ETAM  calls  for  four  major  data  sets. 
These  are: 

Instructional  vehicles  and  their  functional  attributes 

Courses  and  their  instructional  attributes 

Jobs  and  job  tasks  and  their  functional  attributes 

Students  and  their  learning  and  performance  attributes. 

The  search  arguments  developed  by  the  analytic  aspects  of  the  assessment  pro- 
cess will  be  directed  towards  the  Identification  and  retrieval  of  sets  of 
entities  (and  entity  descriptions  and  contexts)  potentially  relevant  to  the 
applicability  of  the  Innovation  or  proposed  change.  Task  5 In  the  ETAM 
procedures  identifies  and  defines  all  of  the  following  terms  and  implicitly 
develops  their  practical  justification. 

Instructional  Vehicles:  Types  and  Functional  Attributes.  The  characterizations 
are  based  on  vehicle  type,  vehicle  properties,  and  on  instructional  context. 

Vehicle  Type 

Instructor 

Static  graphics 

Animated  graphics 

Static  physical  models 

Procedural  trainers 

Symbolic 

Physical  mockup 

Task  simulators. 

This  typology  is  not  intended  to  be  exhaustive,  but  to  represent  a minimum  set 
ofuseful  distinctions.  Each  type  will  have  subset  categories,  and  any  given 
device  or  vehicle  may  overlap  more  than  one  type.  Secondary  characterizations 
may,  of  course,  be  added  to  the  description  of  any  entity. 
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A second  form  of  characterization  of  vehicles  is  by  the  way  In  which  they  manl 
fest  or  Implement  each  of  a set  of  properties.  Properties  of  Instructional 
vehicles  may  be  differentiated  Into  those  primarily  directed  towards  Instruc- 
tional functions  and  those  primarily  directed  towards  utilization  functions. 

The  following  Instructional  functions  are  exhaustive  of  the  variables  that 
comprise  an^  capability  for  learning. 

Instructional  Functions 


Type  of  stimulus  presented  (e.g.,  visual,  auditory,  kinesthetic) 

Type  of  content  presented  (e.g.,  static,  dynamic,  diagrammatic) 

Type  of  presentational  sequence 

Selection  source  for  sequencing 

Type  of  external  control  operated  by  student 

Feedback  presentation  logic 

Response  evaluation  logic. 

A second  class  of  properties  associated  with  Instructional  devices  Is  that  of 
utilization  functions.  No  list  of  utilization  functions  can  be  complete  and 
exhaustive.  A selection  can  be  made,  however,  of  these  functions  which,  taken 
together  as  a set,  make  up  those  generally  most  Important  from  a practical 
sense. 

Utilization  Functions 
Programmability 
Portability 
Multiple  usage 

Operating  simplification  or  usage 
Maintenance 

Updatablllty  In  Instructional  function 
Combining  of  Instructional  capabilities 
Modularization  of  training  content 

Trans latablllty  of  a training  requirement  Into  a training  content. 

Another  characterization  of  Instructional  vehicles  Is  not  an  Inherent  property 
but  Is  based  on  the  class  of  use  for  which  It  Is  applied  or  may  be  Intended. 

In  the  specific  sense,  every  Item  of  a given  Instructional  vehicle  should  be 
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linked  to  every  training  course  and  specific  instructional  objective  in  which 
it  is  used,  so  that  a cross-reference  can  be  made  between  the  instructional 
vehicle  entity  and  the  course  content  entity.  But,  In  addition,  a general 
characterization  of  training  objective  for  which  it  is  generally  used  or  useful 
may  be  useful  for  inquiry  and  search. 

Class  of  Training  Objective 

Reference  knowledge 

Knowledge:  task-specific  and  enabling 

Task  skill  format 
Skill  training 
Procedural 
Tracking. 

A final  category  of  classification  for  the  instructional  entity  is  "Where  Used" 
information.  This  specifies  the  courses  and  course  objectives  and  other  data 
that  enable  the  searcher  to  link  the  device  to  its  specific  applications,  which 
in  turn  may  identify  quantifications  of  how  much  the  device  Is  used,  what  is 
learned  from  or  by  it,  what  the  learning  rate  may  be,  and  so  on.  These  cate- 
gories of  data  are  more  likely  to  be  associated  with  course  data  sets  If  the 
latter  keeps  data  about  training  operations. 

Jobs  and  Job  Tasks.  The  primary  objective  of  the  training  enterprise  Is  to 
take  training  requirements,  based  on  skill  requirements,  and  convert  them  into 
instruction  which  converts  input  student  aptitudes  and  skill  levels  into  on- 
the-job  competences.  Competence  in  jobs  and  tasks  Is  the  ultimate  criterion 
of  training  benefits.  It  Is,  therefore,  the  dependent  variable  to  which  all 
other  variables  contribute  and,  therefore,  should  reference. 

The  characterization  of  job  tasks  should  enable  the  categories  to  be  linked  to 
course  content  specific  to  the  categories,  and  to  Instructional  vehicles  and 
operations.  Each  job  task  categorization  should  not  only  assist  in  formulating 
a search  argument,  but  also  differentiate  a type  of  learning  objective  and/or 
an  instructional  process  or  resource.  The  category  structure  must  optimize 
between  two  conflicting  criteria:  ease  and  simplicity  of  learning,  and  use  vs. 

precision  within  practical  limits  imposed  by  the  procedure  and  the  objective. 

For  present  purposes  in  ETAM,  the  following  structure  is  offered.  Type  of 
skill  is  a characterization  at  about  the  level  of  duty,  and  is  contextual 
rather  than  specific.  It  would  assist  in  identifying  a duty  within  a job. 

Type  of  Skill 

Tracking 

Procedural 
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Decision-Making 

Constructing 

Leadership/Interpersonal . 

The  two  most  Important  sets  of  descriptors  for  jobs  and  tasks  are  those  that 
characterize  task  supports,  and  those  that  characterize  task  structure  elements 
of  task  functions.  Practically  every  task  has  each  of  the  following  kinds  of 
supports  manifest  In  some  way.  The  similarity  or  Identity  of  the  supports  In 
one  task  and  another  tend  to  Indicate  the  degree  of  transferability  of  skill 
from  one  task  to  the  other. 

Task  Support  Characteristics 

Equipment,  objects  used 

Environments 

Tools 

Reference  Information 

Operations  (task  elements) 

Criteria  of  performance:  Time,  errors. 

A supplemental  characterization  of  tasks  Is  In  the  relative  weightings  and 
contexts  of  performance  components,  or  of  elements  or  functions  that  comprise 
the  structure  of  an^  task.  Each  of  these  functions,  as  Identified  below,  has 
somewhat  different  Instructional  requirements,  but  the  Instructional  require- 
ments for  the  given  task  function  are  about  the  same  wherever  that  task 
function  may  appear.  Thus,  one  can  effectively  teach  "Identification  and 
Nomenclature"  with  the  same  Instructional  setup,  whatever  the  job  or  duty  may 
be. 


Task  Structure  Elements 

Goal  projection 

Scan/detect 

Identify 

Interpret 

Procedure  operations 

Track 

Decide 

Construct/plan 


IV-8 


TAEG  REPORT  NO.  12-3 


Motor  performance 
Interpersonal  interaction 
Recall  task-cycle  information 
Recall  enabling  information. 

Every  task  has  some  representation  or  weighting  of  all  of  these  components.  But 
from  the  standpoint  of  skill  learning,  each  task  has  a somewhat  different  mix  of 
these  components,  and  most  tasks  tend  to  have  high  weightings  on  one  or  two.  In 
fact,  the  preceding  structure  suggests  how  tasks  may  be  identified  and  differen- 
tiated from  each  other  for  training  and  for  evaluation  of  performance  purposes. 

If  job  tasks  are  identified  by  preponderant  Task  Structure  Elements,  and  also  by 
descriptions  under  each  of  the  specified  Task  Support  Characteristics,  the 
nature  of  the  skill  is  readily  identified,  transferability  of  skill  from  one  job 
to  another  with  respect  to  this  task  can  be  at  least  grossly  estimated  and  the 
nature  of  the  training  appropriate  to  the  task  is  implied. 

Furthermore,  if  training  objectives  for  skill  learning  are  similarly  coded,  it 
becomes  easy  to  cross-reference  a skill  requirement,  or  an  on-the-job  competence, 
with  the  training  objective.  Not  only  would  this  facilitate  the  prediction  of 
the  effects  of  making  changes  in  some  aspect  of  instructional  technology,  it 
would  also  facilitate  the  development  of  an  archival  data  base  of  experiences 
with  alternatives  in  instructional  technique  applied  to  various  skill  learning 
contexts.  If  these  archives  could  be  accessed  by  this  taxonomy,  ex  post  facto 
research  could  be  conducted  with  historical  data.  The  value  of  such  data  for 
at  least  preliminary  assessment  of  proposed  Innovations  In  technology  could  be 
substantial . 

Student  Characteristics.  The  Navy  has  developed  extensive  classificatory  struc- 
tures  for  characterizing  entrant  personnel  by  tests,  interviews,  and  observa- 
tions. For  purposes  of  this  project,  these  characterizations  may  remain  as 
they  are.  Of  particular  relevance  are  the  patterns  of  tests  and  scores  that 
establish  requirements  for  entrance  to  various  curricula  and  courses  in  Navy 
training.  The  more  demanding  the  requirements  for  entering  and  passing  a 
course,  the  higher  the  relative  manpower  cost  to  the  Navy.  It  is  important 
that  the  data  base  on  students  enable  retrieval  of  the  variables  which  are 
pertinent  to  entrance  to  instructional  programs  and  courses,  and  to  success,  or 
the  lack  of  It,  in  those  programs  and  courses. 

Training  Courses.  Characterizations  .of  skills  and  tasks  provide  one  basis  for 
applying  descriptors  for  courses  aimed  at  skill  learning.  But  there  are  also 
many  courses  directed  towards  knowledges  intended  to  provide  contextual  back- 
ground to  students.  These  instructional  objectives  tend  to  share  greater 
similarities  in  instructional  vehicles  and  techniques  than  are  cormion  to  skill 
learning,  hence,  the  importance  of  making  distinctions  among  subsets  of  such 
material  is  not  so  great.  The  following  varieties  of  reference  knowledge  are 
by  no  means  exhaustive  of  reference  knowledges,  but  are  typically  represented 
in  courses  and  in  objectives  within  courses.  They  may  not  have  the  same  names 
as  are  given.  ^ 
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Type  of  Reference  Knowledge 
System  purposes 
Organizational  roles 
Contexts  of  operations 
Theory  of  operation 
Organization  rules,  constraints 
Nonwork  related  content. 

Idealized  Instructional  technique  could  develop  methods  of  teaching  theory  of 
operations  In  ways  very  clearly  different  from  Instruction  on  these  Other  t§pi§§* 
A tenuous  basis  for  the  categories  does  exist, 

C 

Another  class  of  knowledge  deserves  differentiations,  These  are  knowledges  that 
enable  the  Individual  to  learn  the  skill  mdre  efficiently,  or  to  perform  it, 

Types  of  Enabling  Knowledge 

Operational  goal  criteria 

Nomenclature,  Identifications,  and  locations  of  work  objects 

Procedure  descriptions 

Facts  applicable  to  job  performance. 

Since  enabling  knowledges  can  be  directly  associated  with  skill  learning 
objectives,  they  can  be  evaluated  against  a practical  criterion:  their  contri- 
bution to  learning  the  skill.  It  Is  therefore  useful  to  identify  enabling 
knowledges  In  course  and  course  element  descriptions. 

Skills  of  any  kind  are  learned  In  a series  of  stages,  A given  course,  or 
element  or  unit  within  a course,  may  apply  to  a given  stage  Of  learning.-  The 
content  and  Instructional  technique  and  vehicle  appropriate  to  one  Stage  fl^y 
be  Inappropriate  to  another,  and  in  highly  cost-effective  training  systems,, 
these  distinctions  are  recognized  as  practical*.  It  is,  therefore.  Imported  in 
the  sense  of  being  useful,  to  Identify  a course,  and  especially  units 
courses,  not  only  by  the  kind  of  skill  being  taught  but  also  by  the  skill  level 
being  learned.  The  following  structure  is  used  in  EfAM,  (As  in  all  Of  the 
other  classifications  offered  In  this  section,  the  reader  &an  find  (definition,, 
examples,  and  rationales  for  every  item  In  each  list  in  the  ,€TAM  procedures,, 
Section  III  of  this  report.) 

Stages  of  Learning 

Orientation 

Nomenclature  and  identifications 
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Mediated  procedures 
Mediated  performance 
Automated  performance 

Highly  skilled  performance  in  work  context 
Unusual  task  conditions 
Refresher  learning 

Transfer  of  training  to  new  tasks,  environments. 

The  last  item  is  not  a stage  but  a condition  applicable  to  any  stage,  and  is  a 
realistic  condition  to  identify  for  the  programming  of  training  and  other 
personnel  design  and  decision  situations. 

The  empirical  data  that  should  be  retained  and  accessible  for  any  course  unit 
or  element  that  have  been  characterized  by  the  various  sets  of  descriptors 
specified  here,  are  the  following. 

Empirical  Data  to  be  Retained  for  Courses  and  Course  Elements 

Aptitude  level  for  acceptance  to  course:  should  include  not  only 

cutoff  scores  but  actual  distributions  of  aptitude  scores  of  students 
entering  a given  class. 

Training  time  per  course  and  per  course  element  or  unit  of  instruction. 

Attrition  rate  per  course  and  per  course  element  or  unit. 

Performance  level  achieved  by  student  in  a given  class  and  in  course 
elements  within  the  course;  mean  scores  and  distribution  of  scores. 

Individual  student  performance  scores  should  be  relatable  to  aptitude 
scores,  entering  skill  characteristics,  scores  on  other  courses. 

These  empirical  data  provide  targets  against  which  Innovations  may  be  directed 
and  evaluated.  If  feedback  data  on  students  can  be  obtained  from  on-the-job 
performance,  the  targeting  can  be  more  precise  and  more  reliably  relevant  to 
training  mission  objectives. 

CONCLUSIONS.  It  should  be  emphasized  that  a categorical  structure,  intended  to 
serve  practical  search  and  retrieval  activities  with  human  Interaction  as  part 
of  the  operation,  does  not  require  either  mutual  exclusivity  of  data  categories 
nor  exhaustiveness.  These  may  be  the  desiderata  of  science.  Although  desirable 
in  practical  matters,  they  are  not  essential.  This  lesser  requirement  for 
practicality  is  indeed  fortunate  because,  as  R.  B.  Miller  has  shown  on  logical 
grounds  elsewhere,  it  is  not  possible  to  define  the  meanings  of  "functions," 
"purposes,"  and  "properties"  in  ways  that  are  mutually  exclusive  and  exhaustive. 

These  criteria  can  be  applied  only  to  objects  or  to  the  logical  definitions  of 
sets  of  symbols  treated  only  as  symbols  (i.e.,  as  "objects"). 
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The  relative  ease  or  difficulty  with  which  indexers  can  learn  and  apply  this 
taxonomic  structure  to  Instructional  vehicles,  jobs  and  tasks,  and  courses 
needs  to  be  determined.  At  least  some  of  the  terminology,  and  many  of  the 
concepts,  are  already  In  practice,  although  not  systematically.  It  also 
remains  to  be  determined  how  easily  researchers  and  others  who  wish  to  use  the 
data  base  can  learn  and  use  this  structured  terminology.  Indexing  and  search- 
ing are  reciprocal  activities;  effort  spent  In  Indexing  can  be  retrieved  by 
ease  of  search. 

ETAM  PROGRAM  DESIGN  SPECIFICATIONS 

This  section  sets  down  the  functional  specifications  for  each  of  the  programs 
proposed  for  development  within  the  overall  ETAM  automated  data  systems 
structure.  System  components  exist  at  various  levels  of  specification  and 
design;  some  have  been  taken  to  a preliminary  development  stage.  Some  of  the 
detail  has  been  included  here  to  provide  examples  of  these  various  levels; 
however,  not  every  program  Is  carried  to  the  depth  of  the  examples.  The 
Intent  Is  to  provide  sufficient  guidance  to  a modeler  and  programner  for 
development  of  the  Individual  programs  within  an  Integrated  system's  framework 
It  must  be  pointed  out  that  a major  portion  of  the  design  specifications  Is 
contained  within  the  ETAM  procedures  In  Section  III  of  this  report.  Program 
developers  must  be  thoroughly  familiar  with  the  overall  ETAM  structure  prior  to 
undertaking  modeling  and  programming  tasks. 

OVERALL  SYSTEM  OBJECTIVES.  The  user  Is  a major  concern  In  the  development  of 
any  complex  Interactive  automated  data  system.  The  following  objectives  are 
Intended  as  overall  guidance  to  the  development  process  to  ensure  a high  level 
of  user  orientation  is  achieved. 

1.  Operational  Ease. 

There  will  be  no  requirement  that  the  ultimate  user  be  familiar  with  either 
the  software  or  hardware  design  In  order  to  solve  his  problems  using  the 
system.  This  Implies  an  Interface  with  the  computer  in  a language  which 
is  presently  within  the  user's  repertory,  and  sufficient  prompting  and 
assistance  to  promote  simplicity  In  problem  entry,  as  well  as  In  output 
Interpretation. 

2.  Interactive  Capability. 

Much  of  the  data  entry  Initially  will  be  In  batch  form;  however,  once  the 
basic  analytic  process  has  begun,  a high  degree  of  user  Interaction  will 
be  a part  of  the  overall  design  structure.  The  user  will  be  able  to 
specify  solution  formats,  to  modify  and  override  existing  data,  to  call 
upon  specific  calculation  or  data  manipulation  routines,  or  to  specify 
output  report  formats.  The  assistance  provided  the  user  through  on-line 
prompting  will  be  sufficient  to  permit  rapid  convergency  toward  problem 
solution  and  decision. 

3.  Modular  Design. 

As  the  user  undertakes  the  assessment  process,  additional  calculation  and 
data  manipulation  requirements  may  be  Identified.  The  software  system  will 
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be  designed  in  such  a way  that  existing  routines  can  be  easily  modified 
or  deleted  without  impacting  other  components  of  the  system,  and  so  that 
additional  routines  can  be  developed  and  easily  .incorporated  as  a new 
module  within  the  existing  system  framework.  This  facility  is  necessary 
to  maintain  a "general  purpose"  character  to  the  assessment  process,  so 
that  analytic  processes  unique  to  assessing  certain  types  of  innovations 
can  be  added  as  they  are  identified. 

4.  Data  Base  Maintenance. 

Several  data  bases  have  been  identified  to  interface  with  the  ETAM  pro- 
grams. The  intent  is  to  access  existing  Navy  data  bases,  insofar  as  is 
possible,  so  that  existing  data  base  maintenance  approaches  can  be  used. 
The  objective  Is  to  minimize  the  amount  of  additional  data  to  be  collected 
and  stored.  However,  to  facilitate  the  ETAM  procedures  an  ETAM  Working 
Data  Base  has  been  defined  to  store  ETAM  programs,  project  data,  and  data 
extracted  from  other  Navy  data  bases  to  be  used  In  specific  evaluations. 

ETAM  WORKING  DATA  BASE.  There  are  three  major  sections  of  this  data  base. 

Program  Library.  The  first  section  of  the  ETAM  Working  Data  Base  Is  a program 
library  containing  the  major  programs  and  routines  to  be  called  during  the 
processing  operations.  The  programs  contained  within  this  section  of  the  data 
base  will  be  the: 

♦ 

1.  Description  Structuring  Program 

2.  Range-of-Effect  Program 

3.  Assessment  Program 

4.  User  Interactive  Program 

5.  Data  Base  Maintenance  Program. 

Programs  will  be  stored  as  load  modules  in  a parti tional  data  set.  A direct 
access  storage  device  is  recommended  for  storing  all  programs,  routines,  and 
subroutines  which  are  a part  of  the  program  library;  as  well  as  for  storing 
project  data,  conversion  tables,  abbreviated  records  of  entitles  for  search 
retrieval,  and  extracted  data  from  Navy  data  bases. 

Project  File.  The  second  major  section  of  the  ETAM  Working  Data  Base  will  con- 
tain all  project  data  loaded  initially  using  the  Description  Structuring 
Program,  as  well  as  any  additional  data  developed  as  a result  of  performing  the 
ETAM  operations.  These  data  will  Include: 

1.  Project  Descriptive  Data  (from  Task  1) 

2.  Alternatives  to  the  Innovation  (from  Task  2) 

3.  Risk  Factors  and  Risk  Reduction  Projects  (from  Task  3) 

4.  Decision  Tree  Factors  (from  Tasks  4 and  6) 


IV-13 


TAEG  REPORT  NO.  12-3 


5.  Range-of-Effect  Search  Descriptors  (from  Task  5) 

6.  Benefits  Patterns  (from  Task  5) 

7.  Cost-Benflts  Results  (from  Task  6) 

8.  Financial  Analysis  Data  (from  Task  7) 

In  addition,  abbreviated  records  of  all  courses,  jobs,  and  equipments  will  be 
maintained  as  a part  of  this  data  base.  These  records  will  be  appropriately 
subsetted  (e.g.,  courses  by  phase,  etc.)  and  Indexed  to  permit  search  once  the 
search  descriptors  have  been  formulated  from  Task  5. 

Project  data  will  utilize  an  hierarchical  structure  with  variable  length  records. 
A database  management  system  which  provides  Interactive  capability  (e.g., 
RAMIS®*)  is  recommended  to  facilitate  modifications  to  the  analysis  structure. 
Relationships  between  primary  project  data,  alternatives  to  the  primary  project, 
risk  reduction  project  data,  and  entitles  being  costed  and  aggregated  will  be 
maintained  through  external  tables.  The  components  of  the  project  file;  e.g., 
name,  description,  objectives,  target  applications,  empirical  data,  etc.,  are 
defined  within  the  ETAM  procedures  in  Section  III  of  this  report.  The  ETAM 
procedures  should  serve  as  a guide  in  development  of  project  descriptive  data 
and  risk  reduction  project  data. 

Extracted  Data.  The  third  section  of  the  ETAM  Working  Data  Base  will  serve  as 
a repository  for  the  full  set  of  data  extracted  from  searches  of  master  Navy 
data  bases.  These  data  will  be  maintained  throughout  the  assessment  operation 
and  finally  summarized  and  stored  as  a part  of  the  project  data.  Individual 
hierarchical  data  sets  will  be  established  for  entitles  extracted  from  existing 
Navy  data  bases.  Because  many  of  the  attributes  of  courses,  jobs,  and  equip- 
ments are  subject  to  frequent  change,  once  extracted  they  will  be  used  for 
processing  in  relation  to  a single  proposed  innovation  being  evaluated.  The 
entity  will  be  repeated  in  a separate  data  set  If  It  Is  related  to  more  than  one 
project.  New  extraction  may  be  required  If  the  analysis  spans  a substantial 
time  period  or  If  the  entitles  are  related  to  a major  new  program  which  Is  being 
subjected  to  frequent  change. 

4 

Course  Abbreviated  Data  Base  Structure.  The  primary  use  of  the  abbreviated  data 
bases  Is  for  searching  out  entities  which  match  the  descriptors  formulated 
through  the  assessor's  analysis  of  the  target  application  and  interactive 
prompting  using  the  taxonomies  related  to  the  entity  class  (e.g.,  courses). 

Figure  IV-2  shows  a recommended  structure  for  courses  to  be  maintained  within 
the  project  file  area  of  the  ETAM  Working  Data  Base.  Additional  predevelopment 

study  has  been  recommended  in  Section  I of  this  report  to  develop  Interim 
procedures  for  translating  the  ETAM  nomenclature  Into  descriptions  of  course, 
tasks,  and  Instructional  vehicle  attributes.  This  is  Intended  to  establish 
the  course  Indlx  design  that  will  result  in  a reasonable  economic  tradeoff 
between  storage  requirements  and  retrieval  validity. 


*RAMI$® 


Is  a data  base  language  developed  by  Mathematica,  Inc. 
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LEVEL  1 


CRSNO 


CRSNAME 


CCMM 


LEVEL  2 (REPEATING) 


PHASE 


INDEX 


FIELD  NAMES 

CRSNO:  COURSE  CDP  DESIGNATION 

CRSNAME:  COURSE  NAME 

CCMM*.  COURSE  CONTROL  MODEL  MANAGER 
PHASE : PHASE  OF  COURSE 

INDEX : DESCRIPTOR  TAG  FOR  EACH  PHASE  OF  COURSE 

- STAGE  OF  LEARNING 

- TASK  STRUCTURE  ELEMENT 

- JOB  CLASS 

- CONTENT  TYPE  AND  SUBSET 

FIGURE  IV-2.  EXAMPLE  COURSE  ABBREVIATED  DATA  BASE  STRUCTURE 


LEVEL  1 


RATE/ 

NEC/ 

DESCRIPTIVE 

RANK 

NOBC 

DATA 

LEVEL  2 (REPEATING) 


TASK  ID 


INDEX 


FIELD  NAMES 


RATE/RANK: 

NEC/NOBC: 

DESCRIPTIVE 

DATA: 

TASK  ID’- 


ENLISTED  RATING  AND  GRADE  OR  OFFICER  RANK 

NAVAL  ENLISTED  CLASSIFICATION  OR  NAVAL 
OFFICER  BILLET  CODE 

RATE/RANK  NAME  OR  NEC/NOBC  NAME 
JOB/TASK  DESIGNATION 


INDEX : DESCRIPTIVE  TAG  FOR  EACH  JOB/TASK 

- TYPE  OF  SKILL 

- TASK  STRUCTURE  ELEMENT 

- TASK  SUPPORT  CHARACTERISTICS 


FIGURE  IV-3.  EXAMPLE  JOB  ABBREVIATED  DATA  BASE  STRUCTURE 
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Job  Abbreviated  Data  Base  Structure.  As  with  the  course  ibbrtvlated  data  base, 
the  job  abbreviated  data  base  will  be  used  to  search  job/task  entitles  which 
match  descriptors  formulated  from  the  taxonomies  for  the  .lob/task  entity  class. 
The  recommended  additional  predevelopment  study  In  Section  I of  this  report 
should  provide  the  final  specifications  for  data  base  design.  A preliminary 
recownended  structure  Is  outlined  In  Figure  IV-3. 

Equipment  (Vehicle  Type)  Abbreviated  Data  Base  Structure.  The  Inventory  of 
equipments  used  within  naval  training  would  be  maintained  In  the  third 
abbreviated  data  base.  Descriptive  tags  would  contain  1)  vehicle  type, 

2)  instructional  function,  3)  utilization  functions,  4)  class  of  training 
bbjectlve,  and  5)  where  used  data.  Specifications  for  the  data  base  structure 
are  Intended  to  be  defined  In  the  recommended  additional  predevelopment  study 
Identified  in  Section  I of  this  report. 

DESCRIPTION  STRUCTURING  PROGRAM  (DSP).  The  objective.  Inputs,  processing 
functions,  and  outputs  for  the  DSP  are  outlined  as  follows. 

Objective.  The  DSP  establishes  a new  project  region  within  the  project  file 
section  of  the  ETAM  Working  Data  Base.  It  accepts  the  total  range  of  descrip- 
tive data  on  the  new  project  (proposal  training  Innovation)  and  formats  It  for 
subsequent  processing.  It  also  accepts  major  data  subsets  associated  with  the 
project  which  will  Include:  (l)  experimental  studies  with  descriptions  and 

resultant  data;  (2)  risk  reduction  Implementation  projects  with  descriptions, 
resource  requirements,  schedules,  and  Impacts  upon  overall  project  success; 

(3)  alternative  project  descriptions;  and  (4)  unique  costing  data.  All  major 
data  subsets  will  be  associated  to  the  overall  project  under  analysis  through 
special  relational  tables.  The  table  structure  will  allow  cross-referencing 
of  data  subsets  for  subsequent  updating  and  restructuring  of  selected  data. 

All  data  will  be  edited  by  the  DSP  and  default  data  assignments  will  be  made. 

If  possible,  for  any  missing  data.  Input  to  the  DSP  will  define  the  Initial 
processing  sequence  and  output  requirements. 

Input.  Project  file  Initialization  and  most  updating  will  be  performed  via  card 
Input.  The  major  Input  data  sets  will  be: 

1.  Project  descriptive  data 

2.  Alternative  projects  descriptive  data 

3.  Risk  reduction  project  data 

4.  Cost  models  Input  factors 

5.  Decision  tree  paraemters 

6.  Job  control  Instructions. 

Processing  Functions.  The  DSP  will  accept  Input  data  sets  and  job  control 
instructions.  All  input  will  be  edited  for  missing  and/or  potentially  errone- 
ous data.  Job  control  Instructions  must  Identify  the  manner  In  which  missing 
and  erroneous  data  Input  will  be  handled;  the  DSP  will  perform  the  necessary 
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processing.  The  required  file  management  routines  will  be  called  and  assigned 
to  process  the  input  which  may  be  new  data  or  replacement  data.  Once  data  sets 
have  been  assigned  with  edited  and  default  data  inserted,  a project  table  will 
be  created  which  identifies  all  data  sets  related  to  the  primary  project  being 
proposed. 

Outputs.  Two  major  output  data  sets  result  from  processing  the  DSP.  One  out- 
lines errors  and  default  data  assignments.  The  other  presents  the  requested 
and  generated  job  sequence  instructions  to  the  Executive  Control  System  to 
control  subsequent  processing. 

RANGE-OF- EFFECT  PROGRAM  (REP).  The  objectives,  inputs,  processing  functions, 
and  outputs  for  the  REP  are  outlined  as  follows: 

Objectl ye.  The  two  primary  objectives  of  the  REP  are,  1)  to  create  the  necessary 
set  of  data  base  search  parameters  for  obtaining  the  entities  (e.g.,  courses, 
jobs)  to  which  the  innovation  has  potential  application,  and  2)  to  structure  the 
benefit  pattern  used  by  the  Assessment  Program  in  calculating  the  decision 
variables. 

Input.  Project  descriptive  data  from  the  project  file  will  provide  the  target 
appl i cati ons  for  the  proposed  innovation.  This  will  establish  the  initial 
descriptor  content.  Benefit  estimates  and  the  empirical  data  will  define  bene- 
fit patterns  and  allocation  rules  for  further  modifying  the  search  parameters. 

Processing  Functions.  The  appropriate  abbreviated  data  base  will  be  searched 
based  upon  the  target  application  and  the  descriptor  index  will  be  presented 
for  the  review  of  the  assessor.  A conversion  table  will  translate  nianeric 
indices  to  appropriate  elements  of  the  taxonomies  related  to  the  entity  class 
for  the  target  application.  These  same  tables  will  permit  the  assessor  to 
continually  work  with  English  language  descriptors  during  the  descriptive  index 
refinement  process.  Routines  within  the  User  Interactive  Program  (UIP)  will 
prompt  the  assessor  and  select  taxonomic  menus  to  guide  this  refinement  process. 
Multiple  indices  may  be  developed  if  improvements  of  the  taxonomic  menus  are 
OR'd  together.  Once  the  descriptor  indices  have  been  formulated,  the  appropri- 
ate abbreviated  data  base  will  be  searched  to  obtain  all  entities  which  match. 

Allocation  rules  are  for  the  purpose  of  further  refining  the  list  of  selected 
entities.  For  example,  if  courses  having  a length  of  one  week  or  more  can 
benefit  from  the  proposed  innovation,  this  would  become  an  additional  selector 
factor  when  the  list  of  entities  is  forwarded  to  the  Navy  data  base  controlling 
function  for  extraction  of  the  full  range  of  attributes  associated  with  the 
entities.  The  assessor  must  determine  if  the  additional  selectors  are  to  be 
used  to  limit  the  number  of  entities  extracted,  or  whether  they  should  be 
applied  to  the  total  set  of  extracted  entities  once  they  have  been  loaded  In 
the  extracted  data  section  of  the  ETAM  Working  Data  Base.  This  determination 
would  be  based  upon  whether  the  entitles  eliminated  by  the  selector  would  ever 
be  brought  into  the  Assessment  Program  analysis  from  a change  in  allocation 
rules. 

A final  processing  function  of  the  REP  is  the  formulation  of  the  benefit  pattern 
used  to  input  various  routines  with  the  Assessment  Program.  The  ETAM  procedures 
for  Tasks  4,  5,  and  6 should  provide  sufficient  guidance  on  the  content  and 
format.  Some  benefit  patterns  will  involve  directly  costable  functions  (i.e., 
straight  displaceable  costs)  such  as  in  the  case  where  equipment  is  eliminated 
or  replaced. 
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Output.  The  Range-of-Effect  Program  develops  an  output  tape  containing  a list 
of1  entitles  and  selector  factors  to  be  used  in  searching  existing  Navy  data 
bases  to  obtain  the  full  attribute  set  for  each  entity  matched.  The  second 
output  will  be  a table  of  variables,  degree  of  change.  Importance,  confidence, 
etc.,  which  will  represent  the  benefits  pattern  to  be  used  within  the  Assess- 
ment Program.  Allocation  rules  whlch'were  not  used  In  generating  selector 
factors  on  the  entity  extraction  tape  will  also  be  stored  for  use  by  the 
Assesment  Program. 

ASSESSMENT  PROGRAM  (AP).  The  objectives.  Inputs,  processing  functions,  and  out- 
puts for  the  AP  are  outlined  as  follows. 

Objective.  The  AP  provides  the  data  manipulation  and  computational  capability 
for  generating  costs,  benefit  quantities,  decision  variables,  and  sensitivity 
factors.  These  become  the  key  inputs  to  the  decision  maker.  The  AP  will  have 
the  capability  of  aggregating  and  weighting  entitles  either  for  Inputting  cost 
models  or  resulting  from  processing  by  a cost  model.  Decision  variables  calcu- 
lated on  aggregated  data  will  be  obtained  from  either  a cost-benefits  analysis 
using  a decision  tree  framework,  or  from  a financial  return-on-lnvestment 
analysis. 

Input.  Job  control  instructions  passed  to  the  Executive  Control  System  will 
guide  the  Initial  processing  sequence  for  the  AP,  thus  the  Input  factors 
selected  will  be  a function  of  these  Instructions.  Additional  processing 
Instructions  will  be  provided  by  the  assessor  through  Input  to  the  User  Inter- 
active Program.  Each  routine  (training  cost  model,  job  cost  model,  decision 
tree  generator,  etc.)  will  require  a set  of  data  variables  which  will  be  stored 
In  the  extracted  data  section  of  the  ETAM  Working  Data  Base.  Each  routine  will 
also  have  provision  for  the  generation  or  selection  of  default  data  In  the 
cases  where  Information  Is  missing.  The  individual  requirements  for  the  rou- 
tines with  the  AP  are  outlined  In  the  "processing  functions"  section  which 
follows. 

Processing  Functions.  The  processing  design  should  be  guided  by  the  ETAM 
Procedures,  Task  6 and  7.  The  processing  routines  within  the  AP  are: 

1.  Training  Cost  Model 

2.  Job  Cost  Model 

3.  Equipment  Cost  Model 

4.  Decision  Tree  Evaluator 

5.  Economic  Analysis  Routine 

6.  Sensitivity  Analysis  Routine 

7.  Aggregation  Routine. 

As  previously  mentioned,  each  routine  will  perform  the  data  editing  and  default 
data  selection  for  its  respective  data  sets. 
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Training  Cost  Model.  The  Navy  Training  Analysis  and  Evaluation  Group,  Orlando, 
Florida,  had  developed  a model  for  costing  training  requirements  as  a part  of 
a TECEP  Model  (Training  Effectiveness-Cost  Effectiveness  Prediction  Technique) 
for  evaluating  training .media.  The  basic  logic  of  this  model  was  incorporated 
into  the  Training  Cost  Model  specified  for  ETAM.  A program  listing,  flow- 
charts, and  sample  outputs  from  this  program  are  included  at  the  end  of  this 
section.  Several  features  were  incorporated  into  the  model  which  are  intended 
to  suggest  sophistication  which  may  be  introduced  into  a final  developed 
program. 

1.  Extensive  input  data  editing  and  default  data  specification. 

2.  Linkage  between  the  cost  model  and  the  output  of  a previously  developed 
Training  Process  Flow  Model. 

3.  User  selectable  policies  for  equipment  purchases  and  equipment  depreciation. 

4.  Selectable  output  reports. 

5.  Output  format  design  compatible  with  CRT  display  to  facilitate  any  con- 
version from  the  present  batch  to  an  interactive  system  with  display 
capabi lity. 

Job  Cost  Model.  The  cost  model  recommended  here  provides  a consistency  with  the 
proposed  task  descriptor  discussed  as  a part  of  the  job  abbreviated  data  base. 
Figure  IV- 4 shows  the  Job/Task  Cost  Flow  as  it  relates  to  the  tasks  which  were 
matched  with  the  search  descriptors  formulated  in  the  Range-of-Effect  Program. 
Those  naval  personnel  who  perform  the  tasks  would  have  been  extracted  from  a 
Navy  personnel  data  base  and  stored  in  the  extracted  data  section  of  the  ETAM 
Working  Data  Base.  Allocation  rules  (which  have  not  already  been  applied  in 
limiting  the  entities  extracted)  may  provide  further  selectivity  in  terms  of  "who 
performs  task,"  or  may  be  applied  later  in  the  costing  sequence.  For  example, 
an  Innovation  may  apply  only  to  tasks  which  are  performed  with  a frequency  of 
less  than  one  per  month.  Thus,  certain  tasks  will  be  eliminated  at  this  sub- 
sequent stage.  Figure  IV-4  shows  the  major  cost  determination  paths.  Each 
would  require  that  job  data  beyond  what  is  presently  available  be  maintained. 

Path  1 requires  "time  per  task"  and  "task  frequency"  data  which,  multiplied 
together,  provide  a "change  in  performance  time"  which  can  be  ascribed  to  the 
nnovation.  Performance  time  is  converted  into  numbers  of  personnel  which  are 
now  costable  using  personnel  cost  factors.  A Billet  Cost  Model  presently  under 
development  for  the  Bureau  of  Naval  Personnel  can  be  used  or  will  have  generated 
cost  factors  for  making  the  determinations  of  total  costs  along  this  path.  A 
description  of  the  Billet  Cost  Model  with  a list  of  the  variables  used  in 
calculating  billet  costs  is  reproduced  at  the  end  of  this  section  of  the  report. 

Path  2 provides  for  cost  determination  when  the  innovation  impacts  errors  which 
are  made  in  the  performance  of  a job  or  task.  Errors  can  be  defined  as  any 
deviation  from  criterion  performance.  At  the  present  time,  the  costing  of  this 
path  is  visualized  as  a manual  activity  requiring  a study  of  the  specific  error 
types  affected  by  the  innovation,  and  the  determination  of  costs  associated 
with  the  errors.  , 
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Path  3 is  for  assessing  costs  associated  with  job  support  functions.  This 
would  include  facilities,  tools,  test  equipment,  etc.,  all  of  which  support  the 
job  performance.  Estimating  the  quantities  affected  by  the  innovation  is  again 
visualized  as  a manual  activity,  perhaps  requiring  a specific  study.  Costs  for 
many  of  the  job  support  items  would  be  readily  available  through  the  appropri- 
ate naval  activity  responsible  for  their  procurement  and/or  maintenance. 

The  approach  to  job  cost  modeling  outlined  here  precludes  a need  for  extensive 
parametric  studies  which  would  be  required  if  variables  at  the  next  lower 
level  of  detail  were  introduced.  There  are,  however,  existing  models  that  will 
be  used  for  costing  certain  subsets  of  personnel  of  of  performance  activities. 

An  Air  Force  study*  looked  at  several  alternative  methodologies  and  cost  models 
for  determing  the  cost  of  on-the-job  training.  Thus,  if  an  innovation  was 
targeted  at  changing  this  job  component  (as  in  the  Scenario  Number  One  in 
Section  V of  this  report),  a similar  approach  might  be  taken  in  determining  the 
cost  effects. 

Equipment  Cost  Model.  The  "instructional  vehicle"  route  in  the  Range-of-Effect 
Task  5,  is  taken  when  the  innovation  is  concerned  with  a straight  cost  displace- 
ment effect.  The  data  base  accessed  for  obtaining  attributes  of  equipments 
should  have  cost  factors  available.  The  model  for  determining  equipment  cost 
would  be  similar  to  the  equipment  handling  portion  of  the  Training  Cost  Model. 
Relevant  variables  would  be: 

Equipment  purchase  cost 
Purchase  policy 
Equipment  maintenance  cost 
Life  of  equipment 
Depreci ation  pol i cy. 

Decision  Tree  Evaluator.  The  decision  tree  evaluation  routine  performs  the 
necessary  calculations  required  to  fold  back  the  tree  and  determine  the  value 
of  the  decision  variable.  A program  for  performing  such  an  evaluation  is 
presently  in  use  within  DOD.  The  Decision  Tree  Language  System*  was  used  to 
program  the  standard  decision  tree  developed  for  ETAM.  A copy  of  the  program, 
as  well  as  copies  of  several  program  runs  using  various  features  of  the  program, 
are  contained  at  the  end  of  this  section  of  the  report. 

The  inputs  to  the  decision  tree  evaluation  will  be  able  to  be  entered  directly 
in  the  batch  mode  or  interactively  to  the  User  Interaction  Program.  The  inputs 
to  this  routine  are: 

Outcomes  A through  D 

Success  Probabilities  I,  U through  Z 

Risk  Package  Cost. 


*Samers , B.  N. , Dunham,  A.  D.  and  Nordhauser,  F.  1974.  The  Development  of  a 
Methodology  for  Estimating  the  Cost  of  Air  Force  On-the-Job  Training." 
Report  No.  AFHRL-TR-74-34,  Contract  F41609-72-C-0048,  Hq.  Air  Force 
Human  Resources  Laboratory,  Brooks  AFB,  Texas. 

**A  program  developed  by  SRI  (Stanford  Research  Institute),  Menlo  Park,  Ca, 
is  available  on  the  DARPA  Network  (ARPA/HRRO,  Arlington,  Va) 
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The  program  written  as  a part  of  this  study  does  not  contain  the  facility  for 
Incorporating  more  than  one  risk  reduction  package,  however,  this  addition  can 
be  easily  made. 

Economic  Analysis  Routine.  Since  the  ETAM  procedures  In  Section  III,  Task  7,  of 
this  report  provide  sufficient  guidance  for  this  routine,  and  since  there  are 
undoubtedly  a number  of  computer  programs  available  for  making  these  types  of 
calculations,  no  additional  design  specifications  are  Included  here. 

Sensitivity  Analysis  Routine.  Two  sensitivity  analysis  routines  are  required. 
One  Is  used  with  the  decision  tree  where  the  decision  variable  Is  evaluated  for 
Its  sensitivity  to  each  variable  Included  in  the  tree  outcomes  (A  through  D) , to 
the  success  probabilities,  and  to  the  risk  reduction  package(s)  cost.  As  Is 
outlined  In  the  ETAM  procedures  In  Section  III,  Subtask  06.07,  the  results  of 
the  sensitivity  analysis  will  be: 

Break-even  value 
Decision  change  (from/to) 

Sensitivity  factor 
Probability  factor. 

The  mean  value  and  confidence  range  for  each  variable  are  the  Inputs  to  the 
sensitivity  analysis  routine. 

A second  sensitivity  analysis  routine  (which  can  actually  be  developed  as  a 
subroutine  of  the  first)  will  treat  rate-of-return  and  present  value  decision 
variables  as  the  dependent  variables,  and  will  allow  a sensitivity  calculation 
to  be  performed  using  each  benefit  variable  and  discount-rate  as  Independent 
variables.  A report  similar  to  that  proposed  for  the  sensitivity  analysis  on 
the  decision  tree  should  be  generated.  ETAM  procedures.  Subtask  07.04  should  be 
referred  to  for  design  guidance. 

Aggregation  Routine.  The  assessment  program  must  have  the  facility  for  summing 
the  results  of  evaluating  multiple  entitles.  For  example,  If  the  innovation 
Impacts  fifty  courses,  each  course  would  be  processed  through  the  training  cost 
model,  and  the  results  aggregated  through  this  routine.  Similar  aggregations 
would  be  performed  for  jobs,  equipments,  etc. 

Output.  The  AP  routines  generate  output  data  sets  which  can  be  accessed  by  the 
report  selection,  graphic  report  generation,  and  output  structuring  routines 
within  the  User  Interactive  Program.  The  ETAM  procedures  In  Section  III  contain 
samples  of  most  reports  which  will  result  from  AP  routines. 

USER  INTERACTIVE  PROGRAM  (UIP).  The  objectives,  Inputs,  processing  functions, 
and  output  for  the  UIP  are  outlined  as  follows: 

Objective.  The  purpose  of  the  UIP  is  to  maintain  control  over  all  processing 
functions  and  Input  and  output  both  batch  and  Interactive.  It  serves  as  the 
Executive  Control  for  the  programs,  routines,  and  data  bases.  It  also  controls 
Interaction  with  the  assessor  by  performing  editing,  prompting,  data  retrieval, 
and  output  formatting.  In  this  section,  a general  description  of  the  routines 
within  the  UIP  Is  provided,  however,  final  design  specifications  require  addi- 
tional study  as  recommended  In  Section  I of  this  report. 
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Inputs.  Inputs  to  the  UIP  can  originate  from  several  sources.  Job  control  in- 
structions from  the  DSP  generate  the  calling  sequence  for  processing  the  program 
routines.  These  can  be  overridden  by  terminal  input.  Both  the  job  control 
instructions  and  terminal  inputted  commands  are  analyzed  and  processed  by  the 
Executive  Control  System  within  the  UIP.  Input  data  sets  for  major  routines  are 
accessible  through  the  UIP  in  order  that  data  changes  can  be  made  when  formulat- 
ing "what  if"  type  questions.  The  UIP  also  accesses  these  data  sets,  as  well  as 
any  output  data  sets  from  the  routines,  to  perform  data  retrieval  and  output 
report  structuring. 

Processing  Functions . The  primary  processing  function  of  the  UIP  is  to  handle 
communications  and  control  for  data  handling  and  processing.  An  Executive  Con- 
trol System  containing  a skeleton  monitor  and  communications  monitor  is  the 
focal  point  for  this  function.  The  following  functions  are  also  performed  by 
the  UIP. 

1.  Data  input  and  input  editing.  This  function  is  performed  for  input 
via  the  terminal. 

2.  Prompting.  A number  of  prompting  routines  including  assistance  ("help") 

to  the  user,  will  guide  the  assessor  in  data  input  and  retrieval,  processing 
control,  and  output  structuring. 

3.  Output  reporting.  The  UIP  processes  the  data  sets  outputted  from  the 
various  routines,  structuring  the  data  into  standard  report  formats  as 
outlined  in  the  ETAM  procedures  in  Section  III  of  this  report. 

Output.  The  major  outputs  are  data  exception  reports  from  the  data  editing 
function,  prompting  instructions  to  the  assessor  when  operating  interactively, 
an  interaction  log  which  provides  a record  of  terminal  requests,  graphic  type 
reports  (if  feasible,  based  upon  the  results  of  the  study  recommended  in 
Section  I of  this  report),  and  standard  output  reports. 

DATA  BASE  MANAGEMENT  PROGRAM  (DBMP).  Standard  routines  within  this  program 
perform  initial  data  base  loading,  data  base  editing  and  updating,  and  data 
dumping.  As  a part  of  the  data  base  maintenance  function,  error  and  activity, 
and  maintenance  reports  will  be  provided  to  the  user.  These  programs  operate 
upon  the  elected  data  base  management  system. 


IV-23 


TAEG  REPORT  NO.  12-3 


TRAINING  COST  MODEL 

• Program  Listing 

• Flowcharts 

• Sample  Output  Reports 
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INTEGER  OIGIT 4 201  / 1 • t • 1.4. f .4.T .1. f . 10. 1 1 . It • 11. 14. |9, 14. 

If 

1 IT.tO.lf.tO/ 

14 

LOGICAL  RPTI.RPT2.RPT1.RPT4.IPT9.RPT4.RPTT. LAST 

IT 

OINCNStON  STUO ( 201 .6RA0I 101 .STUONNItOt .SNNRRCItOl .AOOItOI .PSPI20I 

If 

01  MENS  ION  R INSTRI 20 1.RAONlNl 201 t RE QUIP 1 201 .RPACItttOl.l NOMlT 1 201 

1* 

DIMENSION  R SUP PL (20 1 .RNISCI 20) .CSTUOI20 1 .CTRAV (tO).CINStl (101 

to 

DIMENSION  CAONf  NI20 1 .CCOUl  P(20l  • PCQUlPItOI  .P0CPI20I  • AEOillPttOI 

BB 

tl 

DIMENSION  COPNT 120  I .CPACIL 1 tOt .C INONT 1 20) .CfUPPt I 201 

It 

DINCNSION  Cm) SC (201 .OISCNT ( 20) . INPLAT (20) .CGRAOItOI .CSTfNltOt 

tl 

OINENStON  C SPI 20) .NATAI 1001 .RATO  I IOOI.COPTOT 1201 • NCR AO (20 1 

14 

DIMENSION  AO0XI20I 

19 

COMNON/ NOGOP /NOGO . H . MAT  A . NA TR 

1* 

c 

t? 

c progran  initialization 

BB 

tf 

c 

to 

10  AASIN  • 0.0 

>0 

AA04  • 0.0 

II 

tinstr-  0.0 

It 

1 AOMIN-  0.0 

II 

TEOUIP-  0.0 

I* 

TP  AC  1 L-  0.0 

BB 

If 

tnomnt-  o.O 

^B 

14 

TSUPPl-  0.0 

^■F 

IT 

TNISC  - 0.0 

If 

TPEOUP-  0.0 

If 

TAEQUP-  0.0 

40 

DATA  COURSE  / 'NONE • / 

4| 

TPEOUP-  0.0 

BB 

4t 

AACOUM-  0.0 

BB 

41 

PURCMP-  ttfft 

^^F 

44 

OF RE  • tffff 

4f 

RVEQ  - 0.0 

44 

NSTUO  - 0.0 

4? 

OSTUO  • 0.0 

40 

1 STUO  • 0.0 

4f 

ESTUO  • 0.0 

BB 

to 

NTRAV  • 0.0 

W 

fl 

OTRAV  • 0.0 

ft 

ITRAV  • 0.0 

fl 

PTRRV  • 0.0 

14 

NINSTR-  0.0 

ff 

OINSTR-  0.0 

^B 

. 94 

IINSTR-  0.0 

BB 

ft 

P INS  TR-  0.0 

. 

BB 

ff 

NA0N1N-  0.0 

^^F 

to 

OAONII^  0.0 

00 

IAONIN-  0.0 

01 

PAONIN-  0.0 

ot 

NPAClL-  0.0 

01 

OPACIL-  0.0 

4B 

44 

IEACIL-  0.0 

BB 

Ot 

PEACIL-  0.0 

^^F 

00 

NSUNPL-  0.0 

OT 

osupn-  o.o 

40 

ISUNPL-  0.0 

Of 

F SUPPL-  0.0 

TO 

NNISC  • 0.0 

A 

Tl 

ONISC  • 0.0 

■B 

Tt 

INISC  • 0.0 

!■ 

Tl 

PNISC  - 0.0 

^■F 

TO 

NEOUIP-  0.0 

Tf 

Off  QUIP-  0.0 

TO 

Iff QUIP-  0.0 

TT 

PEOUIP-  0.0 

^B 

Tt 

NINO  • 0.0 

m 

TO 

OINO  • 0.0 

to 

UNO  • 0.0 

tl 

PINO  - 0.0 

tt 

OAOR  -0.0 

tl 

SAOR  • 0.0 

04 

NOGO  • 0 

Of 

IGA AD  • 0 

04 

IPSP  - 0 

■A 

OT 

JAOR  • 0 

M 

ARATff  - Mffl. 

00 

■CRATE-  ffftt. 

Of 

ARCVTN.  tffff. 

01 

MSCHIP*  fffft. 

ot 

TSPOSO-  fffft. 

BB 

01 

PISP  • fftft. 

■B 

04 

INTSPO-  fftff. 

VB 

09 

ANTSPO-  fffff. 

^^F 

04 

EOTSPO-  ftftf. 

OT 

LOPEO  • fffff. 

00 

SOPTST-  ffftf. 

tf 

SQPTIN-  ftfft. 

100 

SQPTAN-  fffft. 

jii 

101 

1 MODE t-  ttttt. 

^B 

lot 

PUINO  • ttttt. 

101 

UPOATC-  ttttt. 

104 

IUPSPO-  ttttt. 

100 

SUPS TO-  ttttt. 

104 

NfCSPO-  ttftf. 

I0T 

RSCSTO-  tttff. 

jBk 

100 

STuOSl • ttttt. 

• 

100 

STCSTl-  ttttt. 

• 
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•0/tl/T*  INPUT  lllTItt 


AUTOPLM  CHAAT  NT  - 


CAM  MO 


con  Tim 


no. 

Ill 
lit 
III 
114 
lit 
II* 
Ilf 
II* 
II* 
It* 
III 
III 
It* 
It* 
It* 
It* 
Iff 
It* 
If* 
II* 
III 
lit 
111 
II* 
III 
II* 
IIT 
II* 
II* 
l«* 
1*1 
!** 
1*1 
I** 
I** 
I** 
l*f 
!*• 
14* 
l*« 
III 
lit 
III 
!•* 
Ill 
II* 
Ilf 
II* 
II* 
I** 
1*1 
l»t 
Ml 
I** 
Ml 
I** 
Mf 
M* 
M* 
If* 
Ifl 
If  I 
IT* 
If* 
IT! 
If* 
Iff 
IT* 
IT* 
M* 
Ml 
Mt 
Ml 
M* 
1*1 
II* 
Mf 
!•• 
M* 
1*0 
1*1 
I*! 
1*1 
I** 
I** 
!•* 
l*f 
I** 
It* 
*0* 
t*l 
1*1 
1*1 


t*T 

2*4 

to* 

tIO 

til 

tit 

til 

II* 

III 

tl* 

tlf 

tl* 

II* 

tto 

III 


ITCITf  *****. 

iMtru*  44444. 

AONIL  • 44*44. 

IOUAYC-  *4*4*. 

PCTPCH-  44444. 

CPIOPT-  4*44*. 

C 1*0  • *4*44. 

IVIN  • ****4. 

CHJPP  • *4*4*. 
cm*  - 4*444. 

MAT!  • 4*4*4. 

INAATI-  *4444. 

M • *4*4*. 

UINO*-  4*444. 

■ • 1 

00  10  1*1. to 

CJTUOI I > • 0.0 
CTAAT(I)  • 0.0 
CINlTAt II*  0.0 
CAONINII)-  0.0 
CMUlPIII-  0.0 
COACH  I I I*  0.0 
CINONT'H-  0.0 
ClUPPL(l)-  0.0 

cnisuii  • o.o 

COPTOTtlt*  0.0 
CORAOIIt  • 0.0 
CSTINIII  • 0.0 

cipiii  • o.o 
000*1 1 1 • 0.0 
AMUIKIM  0.0 
ITUOIII  • 0.0 

aooiii  • o.o 
’Mil)  • 0.0 
OINtTOm*  0.0 
AAONINI |l*  0.0 

•iouipui*  o.o 

PIOUIPI 1 1*  0.0 
AtOUIPIIt-  0.0 
OPACIl III*  0.0 
INONMTI II*  0.0 

oium (ti«  o.o 
Mucin  - o.o 
*001(11  • 0.0 
00*01  It  • 4*4*4. 

COPOT  III  • *4**4. 
to  COOT  tout 

c 

C 01 AO  INPUT  DATA 

C 

■AMIIIT/TICMP/  COUAIf .N. AAATI ,TLf MO*. ACAATl .AACTTN.ttSCHOP.TlPOIO. 
1 INTIPO. ANTSPO.IQTSPO.PUACmP «DOPP  ,LOPf Q. I0PT IT , IOP  T (M.SQPTAN* 1*001 V 
t.PUIMO. UPOATt . luPIPOt SUPITD.NlCIPO.NSClTO.Pf IP 
HANOI  I IT/COSTP/ ITllOlL  « ITCIT 1. 1TC1TI « INSTIL .ADNlL.K OUMTC.PCT PCM. CPI 
16PT.C INO.IVIN.CSUPP.CNIS.OAATI. I (MATO 
MANIC I I T/OIMOP /6AA0.C0PRT 
11*0(1, TOCMPI 
MAOll.COITPl 
AlAOf 9.0IN0P) 

•0*011. *000)  APTI.APTI. APTS .RPtA.APTl.APTl.APTT.L AIT 
*400  POANAT  lOil) 

IP  IN.Nt. 4*44*1  00  TO  1* 

c 

C MAO  TPP  NOMC  OATA 

C 

tl  ■* AO  1 10. AOtO.f NO*IO)  COUNS  l.NI , AAAT1I , Til *61 .ACAATl .AACYTI.N6AA0 ( 
1 1 ) . NOAAO( t ) • N0« AO ( I ) . AOOI I 1 1 « AOOK Itl • AOAI (II 
OOtO  POANAT  ( A* .It. P*. I .P4.I. P4.ltPA.lt  111. IP*. 1 1 
IP  tCOuAll.Nff .COUAIt I 60  TO  tl 
N • HI 

AAATI  • AAATII 

UINON  • tlinoi 

ACAATl  • ACAATl 

AACTTN  • ARC  VTl 

00  to  1*1. N 
to  OAAOIII  - NOAAOdl 

c 

C A*  AO  HI  HIM*  OATA  ANAITHI  AMO  OIPAULT  ROUTlNfl 

c 

10  IP  IN.OT.O. AmO.N.LI .tO ) 60  TO  40 
N - tO 
C ACC  NISIN6  (10.11 

40  IP  (AAATI. 60. O..ANO.AAATI.CT. 1.1  60  TO  10 
AAATI  • 0.0 
CALC  *1 11*6  140.1) 

•0  IP  (TCI NON. OT .0. . AM0.TLlM6M.LT.t00. I 60  TO  00 
CALC  AIIIMO  (10,01 

*0  IP  (OCA ATI .00. 0. .AMO.ACAATO.lt. 1 • I 00  TO  TO 
ACAATl-  .MAAATC 
CALL  *1  UNO  1*0.11 

TO  IP  I AACTTN.60.0.. AMO. AACTTN.LT. tOO. I 60  TO  40 
IP  (ACAATl. lO.O.t  60  TO  00 
IP  ITCINOM.CT . 2. ) *0  TO  00 
CALL  NIIANO  (TO. 01 
60  TO  *0 
•0  AACTTN-  0.0 

CALC  NIUNO  ITO.lt 

*0  IP  (tflCHOP. OT. 0.. ANO.VSCMOP.LI.lt. I *0  TO  100 
OlCNOP-  10.0 
CALL  NIUNO  (*0.11 

1*0  IP  IT|P0«0.0«.0.. ANO.TIPOSO.LT. I. I *0  TO  110 
TMOSD-  0.0 
CALL  NIUNO  (IOO.lt 

110  IP  (Pit P.01.0. * ANO.PISP.L*. 1. I *0  TO  It* 

PUP  - 0.1 

CALL  NIUNO  IIIO.II 

ItO  IP  ( I NT IPO. 01.0. .AMO. I NT IPO. L 0.1. I 00  TO  |»0 
INTIPO-  0.0 
CALL  N1MM0  IIIO.II 

110  IP  I ANT  IPtt.O*  .0.  • ANO.ANTSPO.LC.  I • ) 60  TO  140 
ANT IPO—  0.0 
CALC  NIUNO  (I10.lt 
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04/21/79  INPUT  LISTING  AUTOPLON  CHART  SIT  - 


CARO  no 

•••• 

CONTENTS 

222 

140 

IP  lEQTSPO.Gf .0..AN0.EQTSP0.LE.I.I  CO  TO  140 

221 

IP  ICOUNTC. Ml. 99999.. ANO.IQUNTC.Ni. 0.1  CO  TO  190 

224 

IP  ICOPEQ. Nf. 99999. ANO.COPEQ.Nf. 01  CO  TO  190 

229 

EOTSPO-  0.0 

224 

CALL  NISSNO  1140*11 

227 

CO  TO  140 

22* 

190 

CALL  NISSNC  1140*01 

229 

140 

IP  IPUACHf.GE.l.ANO.PUNCMf .LT.IOI  CO  TO  170 

210 

PUICMP-  1 

211 

CALL  NISSNC  1140*11 

212 

1 TO 

IP  lOEPP.CK. I. ANO.OEPP. LT.IOI  CO  TO  ICO 

211 

Of  PP  • 1 

214 

CALL  NISSNC  1170*11 

219 

110 

IP  IIOP EQ.CE.1.ANO.IOPE0.LE.90I  CO  TO  200 

214 

IP  liOTSPO.Nf .0. 1 CO  TO  190 

217 

Ir-  liQUNTC.Nf .99999. . ANO.f OUNTC.NE .0.  1 CO  TO  |90 

210 

LOPEO  - N 

214 

CALL  NISSNC  1 110*11 

240 

GO  TO  200 

241 

190 

CALL  NISSNC  1110*01 

24  2 

200 

IP  ISOPTST.CE.O..ANO.SOPTST.IT.  1000. 1 CO  TO  210 

241 

SQPTST-  0.0 

244 

CALL  NISSNC  1200*1 1 

249 

210 

IP  1 SQP TIN.Ci . 0. . ANO. SQP T| N.L  T • 1000. 1 CO  TO  220 

244 

SQPTIN-  0.0 

247 

CALL  NISSNG  1210*11 

24* 

220 

IP  ISOPTAN. CE.O.. ANO.SQPTAN.LT. 1000. 1 CO  TO  210 

249 

SOPTAN*  0.0 

290 

CALL  NISSNC  1220*11 

291 

210 

IP  IINOOEV. CE.O.. ANO. INOOEV.LT. 1000. 1 CO  TO  290 

292 

IP  IPUINO.CT.O..ANO.PUINO.LE. 1. 1 CO  TO  240 

29S 

IP  IUPOATE.CT.O..ANO.UPOATE.LE.I.I  CO  TO  240 

294 

INOOCV-  0.0 

299 

CALL  NISSNC  1210*11 

294 

CO  TO  290 

297 

240 

CALL  NISSNC  1210*01 

29t 

290 

IP  IPUINO.CT.O. . ANO.PUI NO.LE.l.l  CO  TO  2T0 

299 

IP  IINOOEV. Ni. 0.1  CO  TO  240 

240 

IP  IUPOATE. CT.O.. ANO. UPDATE. Li. 1. 1 CO  TO  240 

241 

PUINO  • 1.0 

242 

CALL  NISSNC  1290*11 

241 

CO  TO  270 

244 

240 

CALL  NISSNC  1290.01 

249 

270 

IP  IUPOATE. CE.O.. ANO. UPOATE.LE. 1.1  CO  TO  290 

244 

IP  1 INODE V.NE.0. 1 CO  TO  2«0 

247 

IP  IPUINO.CT.O. . ANO. PUTNO. LT. I. 1 CO  TO  2>0 

240 

UPDATE-  0.0 

249 

CALL  NI SSNC  1270.1 1 

270 

CO  TO  290 

2T1 

210 

CALL  NISSNC  1270*01 

272 

290 

ip  isupspo.ce.o. .ano.supspo.lt. 1000. 1 CO  TO  100 

271 

SUPSPO-  0.0 

274 

CALL  NISSNC  1290.11 

2T9 

100 

IP  1 SUP ST 0. CE.O. . AnO. SUPSTO.l T. 1000. 1 CO  TO  110 

274 

SUPSTO-  0.0 

277 

CALL  NISSNG  1100*11 

270 

110 

IP  INSCSPO.Cf .0. .ANO. NSCSPO.LT. 1000. 1 CO  TO  120 

279 

NSCSPO-  0.0 

200 

CALL  NT  SSNC  1)10.11 

211 

120 

IP  INSCSTO.CE.b. .ANO. NSCSTO.LT. 1000. 1 CO  TO  1)0 

202 

NSCSTO-  0.0 

201 

CALL  NISSNC  1)20*11 

244 

110 

IP  ISTUOSL. CE.O. .ANO. STUOSL.LT. 90000. 1 CO  TO  140 

219 

stuosl-  0.0 

204 

CALL  NISSNC  1110.11 

2IT 

140 

IP  ISTCSTI. CE.O. .ANO. STCST1.LT. 10000. 1 CO  TO  190 

2M 

STCS T I • 0.0 

It  9 

CALL  NISSNG  1)40.11 

290 

190 

IP  ISTCST2.CE.0. .ANO. STCST2.LT. 10000. 1 CO  TO  140 

291 

STCST2-  0.0 

292 

CALL  NISSNC  1)90*11 

291 

140 

IP  IINSTSL. CE.O. .ANO.INSTSL.lt. 90000.1  CO  TO  1T0 

294 

INSTSL-  0.0 

299 

CALL  NISSNC  1)40.11 

294 

170 

IP  1 AONSL.CE .0. . ANO.AOMSL.LT . 90000. 1 CO  TO  ItO 

297 

AON SL  • 0.0 

291 

CALL  NISSNC  1)70.11 

299 

ISO 

IP  1 E QUNTC. CE .0. . ANO. EOUNTC .Nf . 49444. I CO  TO  400 

100 

IP  IEQTSPO.NE.O. 1 CO  TO  140 

101 

EOUNTC-  0.0 

102 

CALL  NISSNC  1)10.1 1 

101 

CO  TO  400 

104 

190 

call  nissnc  iito.oi 

109 

400 

IP  IPC! PCM.CE .0. .ANO.PCTPCm.LE .1.1  CO  TO  410 

104 

PCTPCH-  0.0 

10T 

CALL  NISSNC  1400,11 

10* 

410 

IP  ICPSOPT. CE.O..ANO.CPSQPT.NE. 99999. 1 CO  TO  420 

109 

CPSOPT-  0.0 

110 

CALL  NISSNC  1410,11 

111 

420 

IP  IC1N0.CE.0.. ANO. CINO.LT. 90000.1  CO  TO  4)0 

112 

CINO  • 0.0 

111 

CALL  NISSNC  1420.11 

114 

410 

IP  ICVIN. CE.O.. ANO. EVIN.LT. l.|  CO  TO  440 

119 

EVIN  • 0.0 

114 

CALL  NISSNC  14)0.11 

117 

440 

IP  ICSUPP. CE . 0. • ANO.CSUPP. NE. 44499 . I CO  TO  490 

111 

CSUPP  • 0.0 

119 

CALL  NISSNC  1440.11 

120 

490 

IP  ICNIS. CE.O.. ANO. CNIS.NE. 49444.1  CO  TO  440 

121 

CNIS  • 0.0 

122 

CALL  NISSNC  1490.11 

121 

4 40 

IP  I0RATI. CE.0..AN0. ORATE. LT. |. 1 CO  TO  470 

124 

OAATI  • 0.10 

129 

CALL  NISSNC  1440. II 

124 

470 

IP  IlNRAli. CE.O. .ANO. INRAIE.LI.I.I  CO  TO  4R0 

127 

INRATE-  0.01 

121 

CALL  NISSNC  1470.11 

129 

410 

IP  (CRAOINI. CE.O.. ANO. CRAOINI.NE. 99499. 1 CO  TO  440 

110 

CALL  NISSNC  1410.01 

111 

490 

IP  ICOPNTINI. CE.O.. ANO. COPNTINI.NE. 49944. | CO  TO  910 

112 

00  900  l-I.N 

111 

900 

COP NT  III  • 0.0 
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M/tt/fl  INPUT  ll  IT  I M MfiPlli  CM— T NT  - 


CMO  NO 

M 

»•# 

CONTVWTft 

m • 

C4LI  NIIINO  1490,11 

n» 

110 

19  IN— 0.10.  I)  40  TO  940 

114 

19  IN.lf.10l  40  TO  919 

111 

IM0099  - N4| 

lftt 

00  919  (• ! MD9I9 • 10 

m 

COPNT(I)  • 0.0 

m 

flf 

HIOIII  • 0 

Ml 

III 

CONTI HOC 

Ml 

c 

Ml 

c 

NO— l 90U4YI0NS  - Tf  CNN  1C  AC  9ACT—  1 

M4 

c 

Ml 

UIROTN  ■ UO—INOM  V«9VINO*40.0 

M4 

CUI—  • U I NOT  MCI  NO 

Ml 

MIN  - CUI—VIN 

M4 

00  910  1*1,0 

Ml 

STUOIII  • 4AA01 1 I/I  1. 0*4114  T|| 

IN 

I4NA0  • 1 IN  A OHM  401  1 1 

III 

AAIIN  • AAf !N*STUO< 1 1 

III 

ST  — < 1 1*  IH«N4***STU0(  lll«(  1.0— .9*40  ATI  1 

191 

SON— Clll*  AC9ATMSTU0II  IVANCTTN 

114 

AOOIII  - ISTUOMNIIUSN— •C(ltl/N|CN09 

Iff 

0409  • 0409  ♦ 1 404 1 1 1-404* (III 

114 

SAOI  • 1409  ♦ 404(11 

If? 

4404  - 44— *4— (|| 

IM 

999(11  • 1 14091 1 )/( 1.0*T 99090  Iff! 1.4*9449 1 1 

Iff 

1919  • 1919*919(1) 

M4 

RIN9T9I 1 I*  919(1 14 IMTS90 

Ml 

UNIT*  • T 1 NIT**t  1N1T*(  I t 

Ml 

■40NINI 1 I*  919I1I44OTS90 

Ml 

T40NIN  « TA— IN*9A0N|N(|I 

M4 

M4UI91II*  919(1  )9«4T190 

Mf 

TI4UI9  * TKOUl 9**I4UI9( I ) 

144 

M4CILIII*  91 9 ( 1 )•  1 S49T IT* I NT  1*0*149T  1 N*  ANT  190*949 T AN  1 

MT 

T9ACIL  • T9ACIL*99ACILtll 

Mf 

(NO— Till-  UINOTMUNOATC 

Mf 

TNOMNT  - TNONNT*  I — T 1 1 1 

170 

UU99LI  II*  9191  I |41U9|90*|TU01 1 I*IUPITD 

171 

T1U99L  • T1U991 *910991 III 

III 

4NISCIII  • 919(1 |4NBCI90*tTUD<l|4MtCITD 

IT! 

J409  • J 409*4011 ( 1 I 

174 

110 

TNI 1C  - TN|1C*INI1CI|» 

•75 

19  I0AO4.1T.0.  I 0404-04— 41-1.) 

Iff 

1404  • 1 10409/ 1404 »♦. 11*100 

•77 

19  II404.6T.9.4N0.J4—  .Nf.O)  CALI  Nltf—  1911,14041 

ITf 

441 IN  • 4411 N/N 

m 

46940  • I6AA0/N 

MO 

4404  • 4409 /N 

101 

A919  • 1919/N 

lit 

4 1 NIT*  • T INIT9/N 

Ml 

4A4NIN  • 1 AON  1 N/N 

M4 

AI4U9  • TC0UI9/N 

Iff 

A99CIL  - T9ACIL/N 

1M 

A1N—T  - TNONNT/N 

MT 

A1U99L  - T1U99L/N 

Ml 

AN  1 1C  • TNI 1C /N 

Mf 

c 

MO 

c 

TN1I  * OUT  INI  C4LCULATCI  (4UI9N4NT  9U9CM4SC  A—  0—4  9CI4TI0N 

Ml 

c 

SCNIOULI!  99—  U1I9  IlLlCTIO  90LICT  — T|—  1 

Ml 

c 

Ml 

c 

- 9— CHAU  901  1C  IIS  ANAL  Till 

m 

c 

Mf 

40  TO  MM.940. 990, 944, 9T4.940, 940,4—, 4101, IUNCN9 

M4 

c 

MT 

c 

90t  ICT  1 - 9U9CHA1C  TOTAL  4V944M 

94NT9 

Iff 

c 

IN  MIST  7IA9  09  I4UI9 

LI9I 

Iff 

ISO 

CON TINUC 

440 

00  994  1*1, N, COM4 

401 

TIN91  • 0.0 

401 

INOIII*  L 0994*1-1 

401 

00  991  J- 1,1*01*1 

404 

T9N91  • AC4UIM  J)*TCN91 

40f 

111 

CONTI NUt 

404 

990019(1)*  TCN91/C0990 

407 

914 

C— TINUC 

400 

60  TO  410 

40f 

c 

IOC  ICT  1 - 9UNCNAM  TOTAL  94NC9NT 

09  Ml  IN— 

410 

c 

90NTS  IN  9I9ST  T9 44  09 

94UI9 

LI9| 

411 

§40 

CONTINUC 

411 

00  944  l*l,N,L09<9 

411 

T9N9I  • 0.0 

414 

INOCll-  L 09(9* 1-1 

♦ If 

00  941  J*l » INOC >1 

414 

19  (TfN9l.LT.9IOU|9(J||  Tf N91*N«0UI9 1 J 1 

417 

941 

C— TINUC 

410 

9ftU|9l||*  TtN9|*9CT9CN 

♦ If 

144 

CON TINUC 

410 

40  TO  410 

4«1 

c 

90t ICT  1 " 9U4CN4S9  U9  TO  AV9IA49 

94NTI 

OUT 

4|| 

c 

NOT  CIC9I4IN6  9—Tft  IN 

T9  09 

9UNCM 

401 

990 

C— TINUC 

4«4 

00  940  1*1 ,N«L09|f 

411 

T4N91  • 0.0 

414 

IN— 11*  L0990*  |-| 

4»7 

M 991  J*l , INOCI 1 

4 If 

TINPI  • 9t4UI9U)*T|N91 

♦If 

991 

CONTINUC 

4M 

9f 4019(11*  T9N9I/L09M 

411 

19  I994UI9I | I .LI.4I4UI9I III  40  TO  994 

4M 

T|N9|  • 91  — 19(11 

411 

INOIII*  10994*1-1 

4M 

00  994  J*t,|N—  XI 

♦If 

19  1 T— 9|  .1.9*  0.  1 60  TO  994 

4M 

19  (TCN9I.AT.4C4UI9UI)  9C4UI9U I-N94UI9UI 

417 

19  IT9N9I.LI.M4U19IJII  9K0UI9 1 J l*Tf  N91 

410 

TCN9I  • TIN9I-9C4UI9U) 

4M 

994 

CONTINUC 

440 

999 

CONTINUC 

441 

60  TO  410 

441 

c 

901  ICT  4 - 9U4CMAM  U9  TO  OCT  09  N49  •— T9  4UT 

♦41 

c 

NOT  IXCII0IN4  9— Tl  IN 

V*  09 

9U4CH 

444 

940 

C—  UNUI 

441 

00  949  1*1 ,N,L09(4 
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COO  NO 


44* 

44? 

44* 

444 

410 

411 

412 
41) 
414 
411 
414 
411 

411 
414 

440 

441 

442 
44) 
444 
441 
444 
447 
441 
444 

470 

471 

472 
4?) 
4 74 
471 
474 
477 
4?i 
474 

410 
4«| 

412 
4«) 
414 

411 
414 
4RT 
4M 
444 

440 

441 

442 
44) 
444 

441 
444 
44? 
44fl 
444 
100 
10! 
102 
10) 
104 
101 
104 
107 
101 
104 
110 
111 
112 
11) 
114 
ID 
) 1 4 
117 
114 
114 
120 
121 
522 
12) 
124 
121 
124 
127 
121 
124 
1)0 
1)1 
1)2 
))) 
1)4 
1)1 
1)4 
1)7 
4)1 

I) 4 

140 
14| 
142 
14) 
144 

141 

144 
14? 

145 
144 
110 
111 
112 

II) 
114 
111 
114 
ii  r 


INPUT  LISTING 


A U TOP  LON  CHART  SIT  - 


• ••• 


CONTENTS 


• ••• 


c 

C 

c 

c 

c 

c 


c 

c 


c 

c 


c 

c 

c 


c 

c 

c 


TEMPI  • 0.0 
INOEXI*  LOFfOM-l 
00  142  2*1* INOf I I 

IP  tTEMP|.LT.*EQU|PI2> I TEMP|*REOUlPUI 
142  CONTINUE 

PtOUlPIII*  TfMP|*PCTPCM 
IP  IPEQulPtll.Lf.REOUlPIIII  CO  TO  141 
TEMPI  • PEOOIPI I I 
INOEXI*  LOPEQ*l-l 
DO  144  2*|. INOEXI 
IP  ITENPi.lE.O. I CO  TO  144 
IP  ITEMPI.CT.REQUIPI2I1  PEOUI PI 2 l-REOUl PI 2 1 
IP  IIEMPI.LE.REQUIPI2II  PEOUlP t 2 l-TEMP I 
TEMPI  • TEMPI- PE QUIPI2I 
144  CONTINUE 
14*  CONTINUE 
CO  TO  420 
170  GO  TO  1)0 
IflO  CO  TO  1)0 
140  CO  TO  1)0 
400  CO  TO  1)0 
410  CO  TO  1)0 

- DEPRECIATION  POLICIES  ANALYSIS 
420  CO  TO  14)0*440*410. 440*470*410, 410,100, 1101  . OEPP 


POLICY  I - DEPRECIATE  LINEARLY  OVER 
LIPf  OP  E0U1PMENT 

4)0  CONTINUE 

TIMP2  • LOP EC 

00  4)2  l-I.LOPEO 
TEMPI  -| 

POEPII!  • TEMPI /TEMP2 
4)2  CONTINUE 
CO  TO  T 10 

POLICY  2 - OEPRECIATE  «Y  SUM-GP-OICI TS 
OVER  LIPE  OP  EOUIPMCNT 

440  CONTINUf 

SUM  • 0.0 

00  442  l*l*LOPEO 
SUM  * SUM* 1 

442  CONTINUE 

POE  PI  I I • ilopeoi/sum 
00  44*  |*2»L0PE0 

POEPII)  • <LOPEO-I)/SUM«POEP<I-I) 

444  CONTINUf 
GO  TO  7)0 

POLICY  ) - SET  DEPRECIATION  SCHCOULC 

TO  PULLY  OEPRECIAVEO  ALL  VfARS 


410 

continue 

00 

412  1*1 

POEPII 

II  « 

412 

I 

i 

u 

CO 

TO 

7)0 

440 

CO 

TO 

4)0 

470 

CO 

TO 

4)0 

410 

CO 

TO 

4)0 

440 

CO 

TO 

4)0 

700 

CO 

TO 

4)0 

710 

CO 

TO 

4)0 

- COST  PACTORS 


710  L • N*  I 

IP  IL.CT.20I  CO  TO  710 
00  740  l-L./O 
GRAOI |l  *0 
740  COPMMLI  • 0.0 
710  CONTINUE 

TOTAEO  • 0.0 
DO  740  |*|, N 

CSTUOIII  • I ISTUOMWII l*SMNRRCI I I I/12I4STU0SL 
CTRAVtll  - STUOIII*ISTCST|*STCST24||.0-.1*AAATEII 
C INST*  I I I • R|  NS  Till  IMNSTSL 
CAOMJNtll.  RAOM|N(||*AOMSL 
TPIOUP  • TPEGUP*PEwUIPIl  I 
IN0EX4*  PEOUlP I 11*0.1 
PEOUlP III*  IN0EX4 
CEOUI Pill*  PEOUI P| II *EOUNTC 
tIPERV  • I *LOPEO~N-l 
IP  ILIPERV.LE.0I  GO  TO  740 
RVEQ  • RVEO*CEOUlPI  I MPOEPILIPERVI 

740  CONTINUE 

TOT  AEO  • TOTAEO’PEOUIPII I 

AE  QUIP  III*  T OT  AEO 
TAEQUP  • TAEQUP*AEQUIPIII 

LORSLT  • l-LOPIG 

IP  ILORSLT.LE.OI  CO  TO  770 
TOTAEO  • TOTAEO-PEOUIPILORSLTI 
AEOUIPIII-  TOTAEO 

770  CIOUIPIII*  CEOUIPt 1 l*AEOUI PII)*COPMTI I I 
CPACILIII*  RPAC1LI I l*CPSOPT 
CIMOMTIII*  | MOMNT III *C I MO 

IP  1 1 .OT. I I CO  TO  771 
CIM0NTI1I*  C I MONT  I I l*CUIMO 

771  CSUPPL III*  RSUPPLI I l*CSUPP 
CMISCI  I I • RMISCI  I l*CM|  S 


- COST  SUMMAM2ATI0N 

DISCNTt II*  I 2.0*0*ATEI/l2.0*< I .0*ORATE I •• I I 

INPLAT I II-  I 2. 0* 1 1.0* INRATE  I •• I I / 1 2 .0* I N*A TE I 

NSTUO  • NS TU0«CSTUOI 1 1 

OSTUO  • OSTUO*CSTUOII|40ISCNTf II 

1ST  UO  • ISTU0«CSTU0I I l*|NPLATI| I 

PSTUO  • PSTU0*CSTU0III*|NPLATI|I40|SCNTII| 

NT* AW  • NT«AW*CTRAV| I I 

OTRAW  • OTRAV*CT*AVlll*OI SCNT 1 1 I 

ITRAV  - ITRAV*CTRAVI||4|NPLATII I 

PTRAV  • PTAAV*CTRAVI 1 1 • 1NPL AT  1 1 1 *0 1 SCNT I 1 | 

NINSTR  • N|NSTR*C INSTRI | I 
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•t/Il/Tl 


INPUT  tltTIM 


autmlm  cnmt  mt  - 


Mi 
»M 
It  0 
MI 
142 
Ml 
M4 
Ml 


m 

tn 

Ml 
1T1 
• Tt 

m 

fit 

ITT 

1T1 

ITt 

110 

m 

MJ 

Ml 

Itt 

Ml 

Mi 

MT 

Mt 

Mt 

MO 

Ml 

Iff 

Ml 

Mi 

ttl 

Mi 

MT 


Ml 

III 

401 


MT 

Mt 

Mt 

010 

til 

til 

ttl 

tit 

til 

tit 

tlT 

tit 

tit 

III 

Ml 

Ml 

til 

tti 

iff 

tit 

tlT 

tit 

tit 

tto 

til 

ill 

til 

tit 

til 

tit 

tlT 

tit 

tit 

tiO 

til 

til 

ttl 

tit 

ttl 

tti 

tlT 

ttl 

tit 

MO 

ttl 

MI 

Ml 

Mt 

til 

Mt 

tlT 

Ml 

Mt 

MO 

Ml 

Ml 

Ml 

Mt 

Ml 

Mt 

MT 

Mt 

Mt 


OINlTt  - DINSTR*C INITR 1 1 I Ml  ICNT  III 

UNIT!  • IlNSTAKlN*TAi||i|NPLAT||l 

P INITR  • P 1NITN*C INITR  f 1 1 4INPL AT  1 1 1 *0 1 ICNT III 

N40NIN  • NA0N|N»CA0N|N4 I I 

040*1  N • 040NIN*CA0NINII  IMItCNTtl  I 

l 40*1  N • 1 AON  I N+C  AON  I Ml  I • *1NPL AT  (II 

MONIN  • PAON|N*CAON|NI|  HINPLAT  IIIMMCM  1 1 1 

NPACIL  • NPACIL*CPAClL<lt 

ORACH  - OPAClL+CPtC IL I 1 I *01  ICNT  til 

IPACU  - IPAClL^CPACIi  I MPINPLATI  I I 

MACH  - PPAC IL+CPACILI  I I *INPL AT  1 1 I *01 ICNT I 1 1 

NSUPPL  • NiUPPl*ClUPPU  II 

OlUP#L  • DSUPPLKIUPPLd  HOI  ICNT  II I 

IIUMC  • 1 f UPPL*C lUPPL  1 1 I *1 NPLAT  (I  I 

PIUMl  • PluPPL»CIUPPLd  HINPLIT  I I IMI1CNTI  | ) 

NNIIC  • NNtSC»CHISCIll 

ONI  1C  - ONI  tC+CM  1 1C  1 1 1 *0 1 KNT ( 1 1 

IN  1 1C  • INIIC*CNItC(tl*l  NPLAT  III 

PNIIC  • PN|  IC*CN  1 1€<  I *i|MPLAT  1 1 1 Ml  ICNT  I 1 1 

IP  d.Nl.NI  GO  TO  TOO 
CMUlPdl«CtOUlPd  t-tVlQ 
C I NON? ( I l*C  I NONT 1 1 l—AVl  N 
TOO  NttUlP  - NiOUlMCfOUltm 

OttUlP  • OlfeJlMtfOUlPI  I t *01  ICNT  I 1 1 

II0U1P  - liQUIPiCCOUIRM  lilNPLATlH 

MtUlP  - PtOUIPiCtOUIPII l*INPLtTttt*OISCNT<  II 

NINO  - NJN04C I NONT I 1 1 

OINO  - 0|ND*C I NONT I I I NO I ICNT III 

IINO  • IINOvClNONTUlilNPLATlII 

PINO  • PINO*CINONT I I liOlICNTl lt*INPlATI 1 1 

COP  TOT  t l ) • CITUOI  I ItCTRAVll  dCINSTAII  t+CAONINI I I*CPACIL  d I 
i »ciNONrui»ciupPim»cNiicn  tiCfouiPiii 

CtRAOlII  • COPTOT I I I/CRAOI I I 
C IT  INI  I I » COPTOTI  I t/STUOin 
TtOCSPIII  • COPTOTII l/PIPII I 

APIOUP  • TPtOUP/N 

AAIQOP  • TAfOUP/N 

NP  AC  li  • NP AC  I L*PACO!T 

OP AC  It  • OP AC  1 1 •PACQIT 

IPACIt  • IPAC IL*P ACOIT 

PPtCIt  • PPAClL«PACOI? 

lNAOC-NSTUO»NTNAV*N|Nira.NAOMlN*NPtClL*NIUPPL»NNItC*MtU!P»NI 
SOtOC-OSlUO«OTAAViOIN|Tt»OAON|N>OPtCU»MUPtl»ON|IC«MNUlPiOI 
llAOC*ISTU0MTAAV«|INlT|»|tON|N«|PACILMIU»PL*|N|*C*lt«U|POI 
IPAOC-PSTUO»PTa»V4P|NSTR«PAON|N*PPACIl •PtuPPL«PN|IC*Pff9UlP*P| 
CNAOC* SNAOC/ IGRAO 
COAOC-iOAOC/IMAO 
tlAOC*t  IAOC/ IMtO 
OPAOC-IPAOC/ICRAO 
UNAOC-SNAQC/ I AAS|N*N>  . 

UO AOC • IOAOC / I AA } I N*N I 
Ult0C*t IAOC/I AAf IN*Nt 
UPtOC«SPAOC/(AAt|N*N> 

PNAOC*INAOC/IPSP 
POAOC-IOtOC/IPSP 
PI AOC •! I AOC/ IP IP 
PPAOC*IPAOC/|PIP 
OtATt  • ORATIilOO. 

INN  All*  IMUTCMOO. 


C MIH  INPUT  TICNN1CAL  PACTORI 

c 

too  IP  (.NOT.flPTII  to  TO  110 

NRITI  It. 10001  C OUR  It  .OAATI.N*  I NR  ATI . I MAO  I I). 1*1,101. 

I ItRAOl I I » I • 1 1*10 1 . ARATI . TIPOIO . TLf NON. Pf IP.MCAATI » 

t f OT IPO 

1000  POftNAT  l'l',2ta, 'COURSE  COIT  ANALYSIS'  ,/,  II.  • INPUT*  • ./.l*. 


• *Pt» 1 » 


1 'COUNIt  NUN RCA  - • .At. 111. *01 ICOUNT  RATI  - 

2 • PftttNT* 'PLANNING  P|R|00  - ',11.171. 

I MNPLATION  RATI  - •,Pt.l,«  PfNCINT*  ,//,  II. 

t 'INPUT  TfCNNlCAi  PACTORIl* v/.ll,*PtNt  YN',«  | 1* 

1 * 1 t | t ? • t 

t * I0'./. II, 'CRAOI-' . 101 ll.lt t *//.!■ .' Pi Nt  *N','  II 

T * II  II  »i  11  It  IT  It  It*, 

• • 20* ,/« 1A , 'ON AOS-'  .10llR*ltl.//.H. 

« 'ATtNITION  RATI  IPCTI  - '.Pl.l.RR. 

1 MINI  ITUO  POI  00  WN  IPCTI  - ' «P  1. 2./.  H. 

A ' TRAINING  t f NCTh  4NNSI  - '.Pl.l.tS, 

• *PCT  CCTRA  ITUO  POSITION!  - *.P). *•/,»■• 

C 'RKVCLI  RATI  IPCT)  - *. Pl.l.RR, 

0 ' IOU I P/ ITUO  POI  RATIO  - '.Pt.ll 

NRITI  I A.  1010)  ARC  TTN.LOPIQ  .NlCMQP  * INOOlY . I NTIPO*  PUINO. 

1 MTIPO.UPOATI.MPTST.lVlN.ltPTIN.IUPIPO.ItPTAN, 

« IUP1T0*  PUNCNP . NIC IPO. OR PP .NIC  I TO 

1010  PORNAT  HR, 


'AVI.  NICVCLl  TIM  (Mil-  '.Pi.l.tR. 
•IIP!  OP  I0UIPNINT  - ',12./, IR. 

'NlfRl  1 CHOCK  OPIRATtl  - *vPi.l,tI, 


COUNIt  MV  Hi i/COORII 
INITR/ITUO  POI  AATIO 
PCT  C0UR1C  MOUIRIM  OCV.- 
AONIN/IIUO  POI  AATIO 
PCT  INITR  NAT.  NAI NT  A I NCO- 
SO  PIIT/lTUOtNl  POI.  - 1 ,Pt. 2. RS 
PCT  INITR  NAT.  INK  VttUfl-  '.PB.2./.1R 
M PRIT /|NITR.  POI.  - '.Pt.I.RA, 

suppl its/s tuoint  PoitnoN-  *.pi.i,/,ia. 


*.P«. l./.li. 

»Pt . 2 • Tl , 

'.Pi.l./.ll. 

,Pt.|,7R. 

PI.1,/,11, 


*10  PIIT/AONIN  POI. 

' SUPPt III/ S TUOINT 
•PUtCHAl*  POLICY 
•NIK/ITUOfNT  position 
' OIPRf C I AT | ON  POLICY 
•N| K /S TUOINT 


*Pt»I, OR. 
'•Pl.i./.ia, 
••II, 111, 
'•Pl.l«/,>R, 
',ii. UR, 
'.Pl.li 


NRITI  INPUT  COST  PACIOtl 


tIO  IP  I.N0T.RPT2)  CO  TO  «»0 

NRITI  It.lOOll  COUR l( . DRAT! ,N, I NR ATI , I COPNT 111,1*1,101. 

* COPNT III, 1*11.20 1, ITUOIt .Cl  NO. ITCST  k .CIUPP. STCIT2, 
2 CNI I.INIIIL.PCTPCN.AONIL.CPIOPT.IOUNTC 

1001  PORNAT  Cl'fltl.  'COUP  It  coil  ANALYSIS', /.U.HmPnTi  ',/.»r, 

I 'COUNIt  NONOta  - ' ,M,  111,  *t  I ICOUNT  RATI  - 'fPt.l, 
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AUTO* 1 0*  CHAM  SIT  - 


CARO  NO  •••• 


CONTENTS 


• ••• 


7*2 

Til 

Tl« 

715 

TiA 

7iT 

719 

7M 

790 
7t  I 
792 

791 

794 

795 
794 
T97 
799 
799 
• 00 
• 01 
•02 
• 0) 
•04 

■ 05 

• 04 
107 

•oa 

•09 

• 10 
•11 
• 12 

• IS 

• 14 

• 15 

• 14 

• 17 

• 1 • 

• 19 

• 20 
•21 

■ 22 

• 2S 

• 24 

■ 25 
■24 

• 27 
•2( 

■ 29 

• SO 

• 31 

• 32 

■ S3 

• 34 

■ 35 

• 34 

■ 17 

■ »• 

• 39 
■40 

• 41 

■ 42 

■ 43 

■ 44 

• 43 
■44 
•47 

■ 4« 

■ 49 
•50 
■31 
■32 
•53 
■54 

• 53 

•34 

■57 

III 

■ 59 

■ 40 

• 41 

■ 42 
■43 
■44 

■ 45 
■44 
■47 
141 
■49 

• TO 

• Tl 

• 72 

■ 73 

■ 74 

• 75 

• T4 

■ 77 
■?■ 

• 79 

• •0 
• •1 
• ■2 

■ ■3 

■ ■4 


4 GOAOC.CI AOC.CFAOC.UNAOC.UOAOC.UIAOC.UFAOC.PNAOC. 

7 POAOC.PI  AOC.PFAUC 

1002  FORMAT  I • l » ,2A*  , •COURSE  COST  ANALVSI  S ' . /•  I X . • I NPUT  » • , / , 31 . 

1 'COURSE  NUMIER  - • . A4 , 231 , *0 1 S COUNT  RATE  - *.F4.l, 

2 • PERCENT' ,/,3X, 'PLANNING  PERlOO  - '.I2.27X, 

3 'INFLATION  RATE  - '.F4.I,'  PERCENT',//,!*, 

A 'OUTPUT  COST  RESULTS  I '• 10K , 

4 'NON-OISC  01 SCOUNTEO  NQH-GISC  01 SCOUNTEO' ,/ , 31 *, 

3 'CONSTANT  I CONSTANT  ■ CURRENT  ■ CURRENT  •'•/•SI, 

4 'STUDENT  SALARIES', 101, 4!2I,E10.0I,/, 31* 

T 'STUDENT  TRAVEL', 12X,4l2X,F10.0t«/«3R, 

• 'INSTRUCTOR  SALARY' ,9X • 4 1 2X ,F 1 0.0) */.3X, 

9 'ADMINISTRATIVE  SAL  ARY • , 3* , 4 1 2« , F 10.01 , /, 3X , 

1 'EQUIPMENT ', 17*. 4 II*. FI  1.0  I, /, 3*. 

A *F  ACUITIES',  14R,4I1R,F1I. 01, /,3Rt 

• 'INSTRUCTIONAL  MA TER  I AL S • , 3R • 4 1 21 ,P |0. Ol , / , 31 , 

C 'SUPPLIES' ,1««, 412*, F 10.01 ,/,!*, 

0 •M|SCELLANE0US**13X*4I2R,F10.0I,//,3R, 

E 'TOTAL  OPERATIONAL',9*,4||rvFII.OI,//,3X, 

F 'COST  PER  GRADUATE* •■l,4|4X, FI. 2l,/,3R, 

G 'COST  PER  STUOENT  I NPUT ' ,3R , 4 1 4R , F • .2  I , /. 3* , 

H 'COST  PER  STUOENT  POSI TION' ,4| 4R,F«.2 It 

C 

C WRITE  ANNUAL  COST  RESULTS 
C 


930  IF  I.N0T.RPT4I  GO  TO  940 

WRITE  14,10041  COURSE. ORATE ,N, INRA TE . I OICI T< 1 1 . I- I • 3 1 , 

1 ICSTUOt I I. I -1,31 .ICTRAVI | I, 1-1 ,51, 

2 ICINSIAI I l,I-l,3I,ICA0M|NI I I , I* 1.3  I, 

3 ICEQUIPI I I, 1-1,31, ICFACtLt 11,1-1,31, 

4 I Cl  MONT  I It, 1-1,31, 1C SUPPL 1 1 1,1-1,31, 

5 I CM  I SC  1 1 I, I -1.51. ICOPTOT II 1,1-1,31, 

4 ICGRAOI 1 I, I- 1,31, ICST IN  I II, I -I ,3 1, 

7 ICSPII 1,1-1,31 

IF  IN.LE.3I  GO  70  940 

WRITE  14,10041  COURSE. ORATE, N, INRATE, lOtGITH 1,1-4,101, 

1 ICSTUOI I I, 1-4,10  I .ICTRAVI I I, 1-4, 101, 

2 1C  I NSTRII 1,1-4.101, ICAOMI Nil  1,1-4*101, 

3 ICEQUIPII t,l-4.10t,ICFAClLII t.l-4,101, 

4 iCt MONTI  I 1.1-4,101, ICSUPPLI 1 1,  1-4,101, 

3 I CM 1 SC  1 1 1.1-4,101, ICOPTOT I 11,1-4,101, 

4 ICGRAOI I 1, 1-4,101 ,ICST I Nil  1,1-4,101, 

7 ICSPI 11,1-4,  101 

IF  IN.LE.10I  GO  TO  940 

WRITE  14,10041  COURSE, ORATE.N, INRATE. IOIGITI I 1.1-11,131, 

1 ICSTUOI I I, I- 11. 131. ICTRAVI I 1,1-11,131, 

2 ICINSTRI 1 1.1-11,131 .ICAOMINI I I. 1-11, 131, 

3 ICEQUIPI I 1, 1-11,131 .ICFACIL 1 1 1.1-11,131* 

4 1C IMDMT I I I, I -11 ,131, ICSUPPLI I I, 1-11, 13  I, 

3 ICMISCI I 1,1-11,131, ICOPTOT 1 1 1,1-11,131, 

4 ICGRAOI I 1,1-11, 131, ICSTINIII,I-1I, 131, 

7 ICSPIII. 1-11. 131 

IF  IN.LE.13I  CO  TO  940 

WRITE  14,1004)  COURSE. DRATE.N, INRATE. IOIGITI I 1*1-14,201 


1004  FORMAT 
1 


ICSTUOI I I, I- 14 .20  I, ICTRAVI 1 1. I- 1 4,201. 

ICINSTRI I 1,1-14.201. ICAOMINI 11,1-14.201. 
ICEQUIPI 1 1 ,1-14, 201 .ICFACIL I I 1. 1 -14,201. 

1C IMDMT I I I, | -14.201 .ICSUPPLI II ,1-14, 201, 
ICMISCI 1 1,1- 14,201 .ICOPTOT I I 1,1-14,201, 

ICGRAOI I 1 ,1-14.201 , ICST INI  11,1-14,201, 

ICSPI I). 1-14.201 

l' ,2«*. 'COURSE  COST  AnALVS I S • , /« 1 1 . * I NPUT < • . / , 3K , 
COURSE  NUMIER  - » . A4 .23* . • 01 SCOUNT  RATE  - '.F4.1, 
PERCENT*,/, 3*,' PLANNING  PERlOO  - *,I2.2TI, 

INFLATION  RATE  - *.F4.1,'  PERCENT ',//, | I , 

OUTPUT  COST  RESULTS!  INOW-OISC  CONSTANT  ftl*,//,32R, 


YEAR  • . 1 2 • • YEAR  '.12.' 

YEAR  ',12.//, 3*. 
STUOENT  SALARIES 
STUOENT  TRAVEL 
INSTRUCTOR  SALARY 
ADMINISTRATIVE  SALARY 
EQUIPMENT 
FACILITIES 

INSTRUCTIONAL  MATERIALS 
SUPPLIES 
MISCELLANEOUS 
total  operational 

COST  PER  GRAOUATE 
COST  PER  STUOENT  INPUT 


YEAR 


.12, 


YEAR 


.12, 


',31  I * , F9.0 I * / , 3* , 
••3I1R,F9.0),/,3K, 
',31 1* , F 9.0  I ,/. 1 K . 
,5tl«,F9.0t./,3K, 
.3IIX.F9.0I./.3X, 
• 3Ui.F9.0l  ,/,!*. 
.31  1K,F9.0I,/,3X, 
,31 1R.F9.0I./.3X. 
,51 IX,F9.0t ,/.3x. 
•31 1K,F9.0I •/• 3X. 
,31  1*,F9.0I,/,3I. 
•3IIK,F9.0),/,3X, 


COST  PER  STUDENT  POSITION  ',31  IR.F9.0I I 
MR  I TE  MISSING  STATEMENT  NUMREHS 


940  IF  I .NOT.IPT  7 1 GO  TO  940 
IF  IK.LE.II  GO  TO  990 
WRITE  14,30001 


X - * - 1 

WRITE  14,30011  IIMATAII  I.MATIII  I),  l-l.KI 

5000  FORMAT  I'l','  MISSING  OATA  ANALYSIS'! 

5001  FORMAT  III, 'MISSING  OATA  COOE  - ',14,41, *CR| I COOE  • • , 14 1 
490  IP  I.NOT.LASTI  CO  TO  10 

CONTINUi 

ENO 

SUBROUTINE  MISSNG  INUMl.lCOOEl 

OIMENSION  MATAI 1 00 1 • HAT • 1 1 00 1 

COMNON/NOGOF/NOGO.X.MATA.NATB 

IF  1 1 COOE . EQ.O I NOCO-I 

MAT  Al X | - NUMI 

MAT ■ I X I - ICOOE 

X - * • I 

RETURN 

ENO 
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• 
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1 AEQUIP 1 11,1  - 
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• 
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■ • 

1 

• 

AC  1 L 1 1 1 , 1 • 4. 
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• 

1 

• 
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• 

1 

• 
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• 
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• 

1 • 4. 
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• 

1 

• 

101. IRNISCII >• 

IREOUIPIII.I  - 

• 

1 

• 

1 ■ 6.  101 

I. SI, 

• 

1 

IOOOT1SI  I NOTE  I 

......... 

LIST  • COURSE, 
ORATE.  N.  INRATE. 
lOICttlll.l  - 
11.131. 
ICRAOMI.I  - 
II. IS). 
ISTUOIll.t  • 

II.  m. 

IAOIfll.1  - 
11.151. 

(PSPIII.I  • 
11.151. 

IR INST  R i 11,1  • 
11.131. 

I R AON INI  I 1 . 1 - 
11. Ml, 

IREOUIPI 1 1. 1 - 


• TRUE 
N .lE.  15  • 


I FALSE 


M 


10007441 


2 WRITE  TO  OEV  2 

2 R 2 

2 VIA  FORMAT  2 
2 IOOS  2 

2 FROM  THE  LIST  2 


I 


1000744 1 | NOTE 


14,201. 

iPEQUlPtll.l  - 
14.20 t , 

lAEQUlPUl.l  • I 
4. 

201 . 1RFACIL III, 
1 • 14. 

201.1 IMOMNT III. 
I • 14. 

201 • lASUPPL 111. 
I • 14. 

20! . I AM  I SCI  1 1, 

I - 14,  201 


1 

1 

1 000 T 541  • to  1 

10007441  | NOTE  16 

1 • FIND. 

NSUP 

1000717)  | NOTE 

05 

1 

• • | 

1 

I 

• • | 

• LIST  - COURSE.  • 

1 

| 

1.51. 

• 

1 

• • TRUE  1 

• ORATE*  N,  INRATE,  • 

1 

1 ‘ 

(AEOUIPIII.I  • 

• 

1 

• N .11.  10  •-  1 

• 1 OICITIII.I  - • 

1 (000TT6I 

1 NOTE 

1.51, 

• 

1 

• *11 

• 14.201.  • 

| • . « • . 

• • • • ( 

IAPACILIII.I  - 

• 

1 

• • 1 1 

• ICRAOdl.l  • • 

1 • PL.  OSUPPL. 

1.  51 . IIHONNT 1 | 1 

• 

1 

• • II 

• 14.201,  • 

1 • 1SUPPL, 

PSUPPL. 

1 • I. 

• 

1 

• «...  1 

• ISTUOIII.I  • • 

1 • NH  | SC » 

0MISC, 

5 1 . 1 ASUPPL (II. 

• 

1 

(FALSE  . 14  . | 

• 16.201*  • 

1 • INI  SC, 

PMISC. 

1 ■ 1 , 

• 

1 

1 . IB  . 1 

• lAOAtll.l  • • 

1 • SNAOC . 

SOAOC. 

Sl.lRMlSCdl,  1 

1 

1 ....  1 

• 16.201,  • 

1 • SIAOC, 

SFAOC, 

1.  51 

• 

1 

1 *40  1 

• (PSPIII.I  • • 

1 • SNAOC , 

SOAOC. 

1 

1 1 

• 14,201.  • 

1 • SIAOC. 

SFAOC. 

| 

1 

1 1 

• IRlNSTRdl.I  - • 

1 • UMAOC. 

UOAOC. 

1 

1 

I000TS5I  | II  | 

• 16.201,  • 

1 • UIAOC. 

UFAOC. 

-------- 

j 

• IRAOHINld.I  • • 

1 • PNAOC , 

POAOC. 

2 WRITE  TO  OEV  2 | 

• 16,201.  • 

1 • PIAOC. 

PFAOC 

2 4 2 1 

• IREOUIPIII.I  - • 

| 

2 VIA  FORMAT  / | 

1 

1 

2 I 005  2 I 

1 

1 

1 

2 FROM  THE  L (ST  2 1 

1 

1 

WRITE  ANNU 

1 

AL  COST 

I 

I 

I 

WRITE  SUMMARY  OF 
ANNUAL  COST  RESULTS 
I 

-14.02*—  >1 


2 1002  2 
2 FROM  THE  LIST  2 

I 

l 

10007741  | NOTE  20 

LIST  • COURSE. 
ORATE,  N,  INRAIE • 
NSTUO,  OSTUO. 
1STU0,  PSTUO. 
NTRAV « OTRAV. 
1TRAV,  FTRAV, 

N| NSTR,  01NSTR. 
1INSTR.  F1NSTR, 
NA0M1N,  DADMIN, 

I AON IN,  PAOHlN, 
NEOUIP.  OEOUIP. 
1E0U1P.  FEQUCP. 
NFACIL.  OFACIL. 
IFACIL.  FFACIL. 
N|MO,  01M0,  1IM0. 
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IV-50 


COURSE  COST  ANALYSIS 


INRUT: 

COURSE  NUMBER  - SEN1 
PLANNING  PERIOD  - 8 

INPUT  TECHNICAL  FACTORS: 
PLNG  YR  1 2 

GRADS-  lOOOO  12000 

PLNG  YR  11  12 

GRADS-  0 0 


DISCOUNT 

RATE  - 

10.0  PERCENT 

INFLATION 

RATE  - 

6.0  PERCENT 

3 

4 

5 

6 

7 

8 

9 

10 

14000 

14000 

13000 

9000 

8000 

5000 

0 

0 

13 

14 

15 

16 

17 

18 

19 

20 

0 

0 

0 

0 

0 

0 

0 

0 

ATTRITION  RATE  (PCT)  - .02 

TRAINING  LENGTH  (WKS)  - 2.0 

RECYCLE  RATE  (PCT)  - .10 

AVE.  RECYCLE  TIME  (WKS)-  1.0 

WEEKS  SCHOOL  OPERATES  - 50.0 

INSTR/STUD  POS  RATIO  - .050 

ADMIN/STUD  POS  RATIO  - .010 

SQ  FEET/STUDENT  POS.  - 9.00 

SQ  FEET/INSTR.  POS.  - 5.00 

SQ  FEET/ADMIN  POS.  - 3.00 

PURCHASE  POLICY  - 2 

DEPRECIATION  POLICY  - 1 


TIME  STUD  POS  DOWN  (PCT)  - .01 

PCT  EXTRA  STUD  POSITIONS  - .05 

EQUIP/STUD  POS  RATIO  - 0.50 

LIFE  OF  EQUIPMENT  - 10 

COURSE  DEV  HRS/COURSE  HR  - 25.0 

PCT  COURSE  REQUIRING  DEV.-  1.00 

PCT  INSTR  MAT.  MAINTAINED-  .20 

PCT  INSTR  MAT.  RMNG  VALUE-  .AO 

SUPPLIES/STUDENT  POSITION-  i.O 

SUPPLIES/STUDENT  - i.O 

MISC/STUDENT  POSITION  - 1.0 

MISC/STUDENT  - 1.0 
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IV-51 


COURSE  COST  ANALYSIS 

INPUT: 


COURSE  NUMBER 

- SEN  l 

DISCOUNT 

RATE  - 

10.0 

PERCENT 

PLANNING  PERIOO 

- 8 

INFLATION 

RATE  - 

6.0 

PERCENT 

INPUT  COST  FACTORS: 

PLNG  YR  L 

2 

3 4 

5 

6 7 

8 

9 

10 

OCMN-  200. 

200. 

200.  200. 

200. 

200.  200. 

200. 

0. 

0. 

PLNG  YR  LI 

12 

13  14 

15 

16  17 

18 

19 

20 

06MN-  0. 

0. 

• 

o 

• 

o 

0. 

o 

• 

o 

• 

0. 

0. 

0. 

STUDENT  SALARY 

(ANNUAL) 

- 11141.00 

COST/HR  OF  INSTR 

MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

228.00 

SUPPLIES  COST 

10.00 

STUD  TRAVEL  IN 

COURSE 

0.0 

MISCELLANEOUS  COST 

- 

0.0 

INSTR  SALARY  (ANNUAL) 

- 16240.00 

PCT 

MAX  EQUIP  PURCHASED 

- 

.90 

ADMIN  SALARY  (ANNUAL) 

- 13500.00 

FACILITY  COST/SQ 

FOOT 

- 

25.00 

EQUIPMENT  UNIT 

COST 

- 5625.00 
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IV-52 


COURSE  COST  ANALYSIS 


INPUTS 

COURSE  NUMBER  - SEN1 
PLANNING  PERIOD  - B 


DISCOUNT  RATE  - 10. 0 PERCENT 
INFLATION  RATE  - 6.0  PERCENT 


OUTPUT  TECHNICAL  RESULTS: 


AVE. 

NO. 

OF  GRADUATES  RQD 

- 

10625.0 

AVE. 

NO. 

OF  STUDENTS  IN 

- 

10886.3 

AVE. 

AVERAGE  ON  BOARD 

- 

452.0 

AVE. 

NO. 

OF  STUDENT  POSITIONS 

- 

478.0 

AVE. 

NO. 

OF  INSTRUCTORS  RQD 

- 

24.0 

AVE. 

NO. 

OF  ADMIN  PERS  RQD 

- 

4.8 

AVE. 

NO. 

OF  EQUIPMENTS  RQD 

- 

239.7 

AVE. 

ANNUAL  EQUIP  PURCHASES 

- 

35.5 

AVE. 

ANNUAL  EQUIP  AVAILABLE 

- 

284.0 

AVE. 

NO. 

SQUARE  FEET  RQD 

- 

4448.7 

AVE. 

INSTR.  MAT.  MAINT  HR  RQD 

- 

400.0 

AVE. 

NO. 

SUPPLIES  RQD 

- 

11365.6 

AVE. 

NO. 

MISCELLANEOUS  RQD 

- 

11365.6 

• • 


• • • • • 
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IV-53 


COURSE  COST  ANALYSIS 

INPUT: 


COURSE  NUMBER  - SEN1 

DISCOUNT  1 

RATE  - 10.0 

PERCENT 

PLANNING  PER IOO  - 8 

INFLATION 

RATE  - 6.0 

PERCENT 

OUTPUT  TECHNICAL  RESULTS: 

YEAR  I 

YEAR  2 

YEAR  3 

YEAR  4 

YEAR  5 

NO.  OF  GRADUATES  RQD 

lOOOO 

12000 

14000 

14000 

13000 

NO.  OF  STUDENTS  IN 

10246. 

12295. 

14344. 

14344. 

13320. 

AVERAGE  ON  BOARD 

425. 

510. 

596. 

596. 

553. 

NO.  OF  STUDENT  POSITIONS 

451. 

541. 

632. 

632. 

587. 

NO.  OF  INSTRUCTORS  RQD 

23. 

27. 

32. 

32. 

29. 

NO.  OF  ADMIN  PERS  RQD 

5. 

5. 

6. 

6. 

6. 

NO.  OF  EQUIPMENTS  RQD 

226. 

271. 

316. 

316. 

1 ?3. 

ANNUAL  EQUIP  PURCHASES 

284.. 

0. 

0. 

0. 

0. 

ANNUAL  EQUIP  AVAILABLE 

284 

284 

284 

284 

284 

NO.  SQUARE  FEET  RQD 

4187. 

5024. 

5862. 

5862. 

5443. 

NO.  INSTR  MAT  MAINT  HR  RQD 

400. 

400. 

400. 

400. 

400. 

NO.  SUPPLIES  RQD 

10697. 

12837. 

14976. 

14976. 

13906. 

NO.  MISCELLANEOUS  RQD 

10697. 

12837. 

14976. 

14976. 

13906. 
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COURSE  COST 

INPUT  a 

COURSE  NUMBER  - SEN1 
PLANNING  PERIOD  - 8 

OUTPUT  TECHNICAL  RESULTS: 

YEAR  6 


NO. 

OF  GRADUATES  RQD 

9000 

NO. 

OF  STUDENTS  IN 

9221. 

AVERAGE  ON  BOARO 

383. 

NO. 

OF  STUDENT  POSITIONS 

406. 

NO. 

OF  INSTRUCTORS  RQD 

20. 

NO. 

OF  ADMIN  PERS  RQD 

4. 

NO. 

OF  EQUIPMENTS  RQD 

203. 

ANNUAL  EQUIP  PURCHASES 

0. 

ANNUAL  EQUIP  AVAILABLE 

284 

NO. 

SQUARE  FEET  RQD 

3768. 

NO. 

INSTR  NAT  MAI  NT  HR  RQD 

400. 

NO. 

SUPPLIES  RQD 

9627. 

NO. 

MISCELLANEOUS  RQD 

9627. 

ANALYSIS 


DISCOUNT 

RATE  - 

10.0 

PERCENT 

INFLATION 

RATE  - 

6.0 

PERCENT 

YEAR  7 

YEAR  8 

YEAR 

9 

YEAR  10 

8000 

5000 

0 

0 

8197. 

5123. 

0. 

0. 

340. 

213. 

0. 

0. 

361. 

226. 

0. 

0. 

18. 

11. 

0. 

0. 

4. 

2. 

0. 

0. 

180. 

113. 

0. 

0. 

0. 

0. 

0. 

0. 

284 

284 

0 

0 

3350. 

2094. 

0. 

0. 

400. 

400. 

0. 

0. 

8558. 

5349. 

0. 

0. 

8558. 

5349. 

0. 

0. 
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COURSE  COST  ANALYSIS 


INPUT: 

COURSE  NUMBER  - SENl 
PLANNING  PERIOD  - 8 

OUTPUT  COST  RESULTS: 

STUOENT  SALARIES 
STUOENT  TRAVEL 
INSTRUCTOR  SALARY 
ADMINISTRATIVE  SALARY 
EQUIPMENT 
FACILITIES 

INSTRUCTIONAL  MATERIALS 
SUPPLIES 
MISCELLANEOUS 

TOTAL  OPERATIONAL 

COST  PER  GRADUATE  773.77 

COST  PER  STUDENT  INPUT  755.20 

COST  PER  STUDENT  POSITION  17167.92 


DISCOUNT  RATE  - 10.0  PERCENT 
INFLATION  RATE  - 6.0  PERCENT 


DISCOUNTED 

NON-DISC 

DISCOUNTED 

CONSTANT  $ 

CURRENT  % 

CURRENT  $ 

28078240. 

48046784. 

34042704. 

14393194. 

24629248. 

17450640. 

2257297. 

3862632. 

2736801. 

375290. 

642186. 

455010. 

1686556. 

1728172. 

1720636. 

644942. 

1103609. 

781943. 

11093. 

17018. 

12989. 

659080. 

1127800. 

799083. 

0. 

0. 

0. 

48105632. 

01157392. 

57999760. 

565.95 

954.79 

682.35 

552.37 

931.88 

665.97 

12556.94 

21184.39 

15139.59 

NON-DISC 
CONSTANT  $ 
30736160. 
19856544. 
3114119. 

517742. 

1732400. 

809749. 

14400. 

909252. 

0. 

65770320. 
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COURSE  COST  ANALYSIS 


INPUT: 


COURSE  NUMBER  - SENI 

DISCOUNT 

RATE  - 10. 

0 PERCENT 

PLANNING  PERIOD  - 8 

INFLATION 

RATE  - 6. 

0 PERCENT 

OUTPUT  COST  RESULTS:  (NON- 

DISC CONSTANT 

$) 

YEAR  I 

YEAR  2 

YEAR  3 

YEAR  4 

YEAR  5 

STUDENT  SALARIES 

4557199. 

5468639. 

6380082. 

6380082. 

5924362. 

STUDENT  TRAVEL 

2336064. 

2803277. 

3270491. 

3270491. 

3036685 • 

INSTRUCTOR  SALARY 

366367. 

439641. 

512914. 

512914. 

476278. 

ADMINISTRATIVE  SALARY 

6091 L. 

73093. 

85275. 

85275. 

79184. 

EQUIPMENT 

1654300. 

56800. 

56800. 

56800. 

56800. 

FACILITIES 

104676. 

125612. 

146547. 

146547. 

136079. 

INSTRUCTIONAL  MATERIALS 

2400. 

2400. 

2400. 

2400. 

2400. 

SUPPLIES 

106971. 

128365. 

149759. 

149759. 

139062. 

MISCELLANEOUS 

0. 

0. 

0. 

0. 

0. 

TOTAL  OPERATIONAL 

9188885. 

9097823. 

10604266. 

10604266. 

9851048. 

COST  PER  GRADUATE 

919. 

758. 

757. 

757. 

758. 

COST  PER  STUDENT  INPUT 

897. 

740. 

739. 

739. 

740. 

COST  PER  STUDENT  POSITION 

20366. 

16803. 

16788. 

16788. 

16795. 

• • 
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COURSE  COST  ANALYSIS 


INPUT: 


COURSE  NUMBER  - SENl 

DISCOUNT  1 

RATE  - 

10.0 

PERCENT 

PLANNING  PERIOD  - 8 

INFLATION 

RATE  - 

6.0 

PERCENT 

OUTPUT  COST  RESULTS:  (NON-DISC  CONSTANT 

S) 

YEAR  6 

YEAR  7 

YEAR  8 

YEAR 

9 

YEAR  10 

STUDENT  SALARIES 

4101479. 

3645758. 

2278599. 

0. 

0. 

STUDENT  TRAVEL 

2102458. 

1868851. 

1168032. 

0. 

0. 

INSTRUCTOR  SALARY 

329731. 

293094. 

183184. 

0. 

0. 

ADMINISTRATIVE  SALARY 

54820. 

48729. 

30455. 

0. 

0. 

EQUIPMENT 

56800. 

56800. 

-262700. 

0. 

0. 

FACILITIES 

94209. 

83741. 

52338. 

0. 

0. 

INSTRUCTIONAL  MATERIALS 

2400. 

2400. 

-2400. 

0. 

0. 

SUPPLIES 

96274. 

85577. 

53485. 

0. 

0. 

MISCELLANEOUS 

0. 

0. 

0. 

0. 

0. 

TOTAL  OPERATIONAL 

6838166. 

6084946. 

3500992. 

0. 

0. 

COST  PER  GRADUATE 

760. 

761. 

700. 

0. 

0. 

COST  PER  STUDENT  INPUT 

742. 

742. 

683. 

0. 

0. 

COST  PER  STUDENT  POSITION 

16840. 

16858. 

15519. 

0. 

0. 
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JOB  COST  MODEL 

• Description  of  Billet  Cost  Model* 

• Billet  Cost  Model  Variables 


^ReproH^d  from  documentation  on  the  Navy  Military  Manpower  Billet  Cost 
Personnel  1fe  CyCle  Plann1ng  PurP°se  provided  by  the  Bureau  of  Naval 
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DESCRIPTION  OF  BILLET  COST  MODEL 

INTRODUCTION.  The  Chief  of  Naval  Personnel  has  developed  the  data  in  this  pub- 
lication for  use  in  computing  the  life-cycle  costs  for  major  Navy  weapon  sys- 
tems (ships,  aircraft,  etc.).  These  data  are  provided  to  civilian  contractors 
competing  for  and/or  participating  in  the  formulation  of  weapon  system  design 
and  procurement,  and  are  used  by  the  Navy  and  the  Office  of  the  Secretary  of 
Defense  in  the  evaluation  of  contractor  proposals.  In  addition,  they  are  used 
for  many  other  management  purposes,  wherever  the  projected  costs  of  military 
manpower  are  significant. 

Manpower  billet  costs,  as  defined  by  the  DOD  Manpower  Cost  Model  Study  Group, 
include  the  total  cost  to  the  U.S.  Government  of  manning  an  established  opera- 
tional military  billet,  or  a military  billet  which  would  be  established  within 
a proposed  system.  These  costs  are  presented  as  annual  costs  for  both  officer 
and  enlisted  personnel  and  are  used  in  costing  the  life-cycle  of  the  system 
under  consideration.  Manpower  billet  costs  differ  significantly  from  appro- 
priation-oriented personnel  costs  which  are  used  for  budget  planning  purposes, 
since  they  contain  values  which  are  not  taken  into  account  in  appropriation- 
oriented  costs.  For  example,  the  values  established  within  the  formula  for 
the  purpose  of  costing  the  "down  time"  of  personnel  in  service  schools,  and 
the  values  developed  to  reflect  the  effect  of  "continuance  rates"  are  not  con- 
tained in  personnel  cost  data  used  for  budget  plannning  purposes. 

Manpower  billet  cost  data  contained  in  this  publication  are  not  complete  costs, 
but  reflect  mainly  the  cost  to  the  Navy  and  the  Department  of  Defense. 

Manpower  Billet  Costs  are  developed  for  both  officer  and  enlisted  personnel. 

The  costs  are  computed  in  a manner  which  results  in  an  "operational  billet  cost 
per  year"  for  all  operational  billets  in  the  Navy  with  the  exception  of  flag 
billets  and  pjygrade  E-l  billets.  In  order  to  compute  these  costs,  the  Chief 
of  Naval  Personnel  has  developed  a Billet  Cost  Model.  Basic  pay  data  included 
in  this  publication  are  based  on  January  1973  Pay  Tables.  A summary  descrip- 
tion of  the  major  cost  elements  within  the  Billet  Cost  Model  is  contained  in 
Section  III.  This  general  desription  is  applicable  to  both  officer  and  enlist- 
ed billets,  and  is  provided  as  an  explanation  of  billet  costing.  It  is 
presented  In  terms  of  enlisted  billets  to  simplify  the  explanation. 

It  is  planned  to  revise  the  publication  annually  to  include  changes  In  the 
cost  elements  brought  about  by  military  pay  raises,  changes  in  training  costs, 
etc. 

DISCUSSION.  Manpower  costs  are  accrued  from  operational  billets.  An  opera- 
tional billet  is  best  described  as  a job  position  which  is  being  filled  at  all 
times  by  personnel  in  an  operational  status.  All  other  personnel  such  as 
students,  patients,  prisoners,  or  personnel  in  a transient  status  are  not 
occupying  operational  billets.  Manpower  related  costs  incurred  during  opera- 
tional times  are  considered  in  the  model  formulae  as  "UP"  costs,  and  costs 
incurred  during  nonoperational  times  are  known  as  "DOWN"  costs.  DOWN  costs 
Include  pay  related  cost,  (when  incurred  in  the  DOWN  status)  and  an  apportion- 
ment of  support  and  certain  one-time  occurring  costs,  such  as  procurement 
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(recruiting)  costs,  based  upon  the  length  of  the  DOWN  time.  The  concept  is 
that  DOWN  costs  represent  a manpower  resource  investment  cost  which  the  Navy 
must  make  in  order  to  obtain  future  operational  service.  Therefore,  all  DOWN 
costs  are  amortized  over  future  operational  billet  years. 

One  major  element  of  cost  to  the  U.S.  Government  which  must  be  apportioned  to 
operational  billets  Is  that  related  to  retirement  costs  which  occur  after 
career  men  finish  their  operational  duty.  The  total  cost  of  retirement  per 
individual  (called  the  principal  value  of  retirement)  is  developed  on  the 
basis  of  life  expectancy  data.  This  cost,  computed  on  the  basis  of  average 
age  at  the  time  of  retirement  from  specific  grades.  Is  distributed  by  the 
model  back  (In  time)  over  the  operational  billet  years.  Several  methods  of 
distributing  this  cost  have  been  developed  and  tested.  The  one  found  to  be 
the  best  for  billet  costing  purposes  and  used  In  this  model  is  the  method 
which  distributes  the  principal  value  of  retirement  as  a fixed  percentage  of 
active  base  pay.  This  method  was  approved  for  billet  costing  by  CNO, 

0P-90E1 1/pas  Ser  620P90  of  22  April  1969. 

A primary  factor  characterizing  manpower  billet  costs  Is  that  these  data  are 
to  reflect  the  total  cost  to  the  U.S.  Government,  not  just  to  the  Navy  or  the 
Department  of  Defense,  of  manning  an  established  operational  military  billet, 
or  a military  billet  which  would  be  established  within  a proposed  system. 

The  costs  developed  by  the  model  results  in  an  "operational  billet  cost  per 
year"  for  all  operational  billets  in  the  Navy  except  flag  billets  and  paygcade 
E-l  billets. 

In  developing  life-cycle  costs,  the  Model  views  the  manpower  picture  as  a 
flow  of  men  through  each  rating  as  well  as  through  each  billet.  They  are 
procured,  training,  utilized  and,  as  time  passes,  they  are  lost  through  non- 
reenlistment,  death,  retirement  or  other  attrition  factors.  Manpower  costs 
are  thus  computed  from  initial  procurement  to  the  end  of  retirement  and 
charged  to  an  active  duty  base  of  twenty- five  operational  billet  years. 

Each  rating  is  viewed  by  the  Model  as  a flow  of  men  through  a pipeline.  The 
pipeline  Is  divided  into  year-long  intervals  and  all  costs  incurred  by  the 
rating  in  each  interval  are  noted.  During  the  interval,  some  men  are  lost 
through  various  leaks  in  the  pipeline,  e.g.,  nonreenlistment,  change  in  rating, 
death,  etc.  No  attempt  Is  made  to  analyze  these  leaks.  The  only  criterion  of 
consideration  Is  the  quantity  of  men  entering  the  next  Interval.  The  relative 
number  of  men  Is  considered  because  the  actual  number  affects  only  the  total 
cost  and  not  the  cost  per  man/billet.  Accordingly,  the  flow  Is  normalized 
such  that  one  man  retires  at  the  end  of  the  career  span,  i.e.,  twenty-five 
years.  Thus,  normalization  is  a function  of  the  flow  rate  in  a given  rating 
. and  varies  with  the  different  ratings.  It  dictates  that  the  number  of  men 
which  must  be  Introduced  into  the  pipeline  the  first  year  to  produce  one  man 
retiring  at  the  end  fluctuates  significantly. 

In  summary,  the  Billet  Cost  Model  is  quite  flexible.  It  accounts  for  the  many 
variables  which  make  up  life-cycle  operations  costs.  It  determines  which  cost 
element  applies  to  all  ratings  by  year  of  service  or  grade  or  which  cost 
element  applies  to  a particular  rating.  As  each  element  is  identified  by  a 
card  in  the  model,  a value  Is  added  to  each  appropriate  rating  packet  as 
constant  $ by  year  or  constant  $ by  grade.  A detailed  list  of  cost  Items 
presently  being  used  is  shown  in  the  Billet  Cost  Model  Data  Summary. 
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The  preliminary  computations  to  obtain  the  input  data  cards  for  the  model  are 
external  to  the  model.  The  model  will'  accept  numbers  up  to  six  digits.  The 
precomputations  and  source  of  data  have  been  kept  as  a matter  of  routine  so 
that  future  cost  changes  can  be  incorporated  on  an  "as-required"  or  "as-occurred" 
basis.  Moreover,  there  is  no  fixed  number  of  rating  packets  that  must  be  run 
at  one  time.  This  allows  for  experimentation  with  a single  rating  packet  or 
a group  of  rating  packets.  A hypothetical  rating  can  be  created  by  defining 
a new  rating  XXX  and  preparing  a cost/statistical  profile  the  same  as  a regu- 
lar rating.  The  hypothetical  rating  can  be  run  by  itself  or  with  a regular 
rating  to  obtain  a comparison  of  results.  Similarly,  either  the  enlisted  or 
officer  decks  can  be  run  separately. 

MANPOWER  BILLET  COST  MODEL  DATA  DESCRIPTION.  This  section  contains  a brief  sum- 
mary description  of  some  representative  cost  elements  within  the  Billet  Cost  Model. 
While  it  is  not  a complete  list,  it  indicates  the  range  of  data  used  to  develop 
billet  costs.  Since  actual  values  are  subject  to  significant  change,  only  gen- 
eral data  are  given. 

Grade  Costs 

Sources:  Budget  FY  1972  Military  Personnel,  Navy 

Budget  FY  1972  Operations  & Maintenance,  Navy 

These  costs  include  the  following,  computed  for  each  officer  and  enlisted  pay 
grade: 

Family  Separation  Allowance 

"Q"  Allotment 

Quarters  Allowance  BAQ 

Sea  and  Foreign  Duty  Pay 

Tuition  Assistance  and  Dependent  Schools. 

Annual  Costs 

Sources:  Budget  FY  1972  Military  Personnel,  Navy 

SECNAV  Task  Force  on  Navy/Marine  Corps  Personnel 
Retention,  Vol . X 

Includes  annual  costs  such  as: 

Clothing  Allowance 
Death  Gratuity 
Commissary 
Exchange 
Medical  Costs 

Subsistence  Allowance,  etc. 

Base  Pay 

Source:  DOD  Military  Pay  and  Allowances  Entitlement  Manual  (as  changed). 

Hazardous  Duty  Pay  (Aviation  and  Submarine) 

Source:  DOD  Military  Pay  and  Allowances  Entitlements  Manual. 
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Continuance  Rates 


Source:  BuPers  Master  Personnel  Tapes. 

Average  Length  of  Service  to  Advance 

Source:  Pers-A12  for  enlisted 

Pers-All  for  officers 

Constant  Dollar  Cost  by  Grade 

Sources:  Navy  and  Marine  Corps  Military  Personnel  Statistics,  NAVPERS  15658 
Proficiency  Pay  List,  Report  E-170 
Manual  of  Navy  Enlisted  Classifications,  NAVPERS  15105 
Budget  1972  Military  Personnel,  Navy 

Includes  cost  Items  that  can  be  Identified  by  Rating  and  Grade,  such  as: 

Proficiency  Pay 

Hazardous  Duty  Pay  (non-crew) 

Flight  Deck  Duty  Pay. 

Transportation  Costs 


Source:  Budget  FY  1972  Military  Personnel,  Navy 

These  costs  are  divided  into  four  categories:  accession  travel,  change  of  sta- 

tion travel,  training  travel  and  separation  travel.  Computations  are  done  sep- 
arately for  enlisted  and  officer  personnel  and  data  are  included  only  where  ap- 
propriate. 

Constant  Dollar  Cost  by  Year  of  Service 

Sources:  DOD  Military  Pay  and  Allowances  Entitlements  Manual 

BuPers  Instruction  1133.18C 
Pers-A12 

Includes  cost  items  that  can  be  identified  by  Rating  and  Grade  and  Year  of 
Service,  such  as  Reenlistment  Bonus  and  Continuation  Pay. 

Median  Length  of  Service 

Sources:  BuPers  Report  E 159H  Report  of  Enlisted  Personnel  on  Active  Duty 

Pers-All 

Officer  Management  Simulation  Model 
School  Training  Costs 

Sources:  Computed  by  Occupational  Standards  Division,  Naval  Personnel  Program 

Support  Activity,  Washington,  D.C.,  for  enlisted. 

For  officers,  data  procured  from  Pers-All  and  0P-992E. 

Includes  all  costs  related  to  school  training.  Including  procurement  costs  as 
a part  of  Initial  training. 


% 
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INDEX  OF  COST  ITEMS 

PROGRAM 

ALL  ENLISTED 
♦BP  ^HP  *YR  *GR 

EACH  RATING  PACKET 
PRO  REN  SCH  TRV 

ALL  OFFICERS 
•BP  *HP  *YR  *GR 

EACH  RATING  PACKET 
PRO  REN  SCH  TRV 

BASE  PAY  TABLE 

X 

X 

CLOTHING  ALWS  - INITIAL 

X 

X 

- BASIC 

X 

-STANDARD 

X 

-SPECIAL  E-7 

X 

COMMAND  AND  ADMINISTRATION 

X 

X 

COMMISSARY 

X 

X 

DEATH  GRATUITY 

X 

X 

DENTAL  PAY 

X 

DEPENDENT  SCHOOL 

X 

DISABILITY  PAY 

X 

X 

EXCHANGE 

X 

X 

FAMILY  SEPARATION 

X 

X 

FICA  • GOVT.  CONTRIBUTION 

X 

HAZARDOUS  DUTY  • AIRCREW  & SUB 

X 

X 

HAZARDOUS  DUTY  - NON-CREW 

X 

X 

INTEREST  ON  DEPOSITS 

X 

X 

INSURANCE  - FHA  HOUSING 

X 

X 

MEDICAL  COSTS 

X 

X 

MEDICAL/VETERINARIAN  PAY 

X 

MESSING/SUBSISTENCE  ALWS 

X 

X 

OVERSEAS  STATION  ALLOWANCE 

X 

X 

PERSONNEL  PROCUREMENT 

X 

X 

PROFICIENCY  PAY 

X 

• 

"Q"  ALLOTMENT 

X 

QUARTERS  ALWS 

X 

X 

QUARTERS  - SINGLE  (BARRACKS) 

X 

X 

QUARTERS  - MARRIED  (HOUSING) 

X 

X 

RECREATIONAL  FACILITIES 

X 

X 

REENLISTMENT/CONTINUANCE  PAY 

X 

X 

RESPONSIBILITY  PAY 

X 

RETIREMENT  PAY 

X 

SCHOOL  TRAINING 

X 

X 

SEA  AND  FOREIGN  DUTY  PAY 

X 

X 

SERVICEMANS  GROUP  LIFE  INS 

X 

X 

SETTLEMENT  COSTS 

X 

SEVERENCE/READJUSTMENT  PAY* 

X 

TRAVEL/TRANSPORTATION 

X 

X 

TUITION  AID 

X 

X 

UNEMPLOYMENT  COMPENSATION 

X 

X 
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DECISION  TREE  EVALUATOR 

• Tree  Format  With  Nodes  and  Variable  Designators 

• ETAM  Decision  Tree  Format 

Program  Listing 
Run  CTREE 
Run  Command 
Gamma  Command 
List  Command 
Gamma  Command 
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IMPORTANT  OUTCOMES 


A SAVINGS/BENEFITS 
ACCRUE 


B COSTS  POSSIBLE 
LIABILITIES  RESULT 


C COSTS  ACCRUE 


O NEUTRAL  OUTCOME 


A SAVING  S/BENEFITS 

ACCRUE 


• COSTS  POSSIBLE 
LIABILITIES  RESULT 


C COSTS  ACCRUE 


O NEUTRAL  OUTCOME 


A SAVlMGSftENEFITS 
ACCRUE 


B COSTS  POSSIBLE 
LIABILITIES  RESULT 


C COSTS  ACCRUE 


O NEUTRAL  OUTCOME 


ETAM  DECISION  TREE  FORMAT  - NODE  AND  VARIABLE  DESIGNATORS 
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ETAM  DECISION  TREE  FORMAT* 


.1  C:  ETAM  DECISION  TREE  FORMAT 

.2  GENERATION 

.3  TYPE  PR(2) ,TE(2) , DE(2) 

.4  NODE  DECIDE.DE 

.5  VAL( 1 )=6HDECIDE 

.6  IVAL(2)=1 

.7  NEXT  NORISK 

.8  NEXT  RKPKG1 .COST1 

.9  END 

1.  NODE  NORISK.DE 

1.1  VAL(1 )=6HN0RISK 

1.2  IVAL(2)=2 

1.3  NEXT  ACPTOO 

1.4  NEXT  - ,D 

1.5  END 

1.6  NODE  ACPTOO, PR 

1.7  VAL( 1 )=HACPTOO 

1.8  IVAL  (2)=3 

1.9  NEXT  IMPLOO.PU 

2.  NEXT  -.l.-PU.C 

2.1  END 

2.2  NODE  IMPLOO.PR 

2.3  VAL(1 )=6HIMPL00 

2.4  IVAL(2)=4 

2.5  NEXT  -,PV,A 

2.6  NEXT  -.l.-PV.B 

2.7  END 

2.8  NODE  RKPKGl.PR 

2.9  VAL( 1 )=6HRKPKG1 

3.  I VAL ( 2 ) =5 

3.1  NEXT  PSUCC1  ,PI1 

3.2  NEXT  PFAIL1.1.-PI1 

3.3  END 

3.4  NODE  PSUCCl.DE 

3.5  VAL(1 )=6HPSUCC1 

3.6  IVAL  (2)=6 

3.7  NEXT  ACPT11 

3.8  NEXT  - ,D 

3.9  END 

4.  NODE  ACPT11 ,PR 

4.1  VAL( 1 )=6HACPT1 1 

4.2  I VAL( 2 ) =7 

4.3  NEXT  IMPL11.PW1 

4.4  NEXT  -.l.-PWl  ,C 

4.5  END 

4.6  NODE  IMPL11.PR 

4.7  VAL(1 )=6HIMPL11 

4.8  IVAL(2)=8 

4.9  NEXT  - ,PX1  ,A 

5.  NEXT  -.l.-PXl.B 

5.1  END 


*Developed  using  SRI  (Stanford  Research  Institute)  tree  language 
system,  SRI,  Menlo  Park,  California  94025. 
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5.2 

5.3 

5.4 

5.5 

5.6 

5.7 

5.8 

5.9 
6. 
6.1 
6.2 

6.3 

6.4 

6.5 

6.6 

6.7 

6.8 

6.9 

7. 

7.1 

7.2 

7.3 

7.4 

7.5 

7.6 

7.7 

7.8 

7.9 

8. 
8.1 
8.2 

8.3 

8.4 

8.5 

8.6 

8.7 


NODE  PFAILl.DE 
VAL  ( 1 )=6HPFAIL1 
IVAL  ( 2) =9 
NEXT  ACPTIO 
NEXT  -,C 
END 

NODE  ACPTIO, PR 
VAL( 1 )=6HACPT10 
IVAL  ( 2 ) =1 0 
NEXT  IMPLIO.PYl 
NEXT  -.l.-PYl.C 
END 

NODE  IMPLIO.PR 
VAL(1)=6HIMPL10 
IVAL(2)=11 
NEXT  -.PZl.A 


NEXT  -.1.-PZ1.B 
END 

C:  INPUT  STATEMENTS 


WRIT 

READ 

READ 

READ 

READ 

READ 

READ 

READ 


(5,84) 

5.85) A 

5.85) B 

5.85) C 

5.85) D 

5.85) C0ST1 
5,85  PI  1 


5,85  PU 
READ(5,85)PV 
READ  5,85  PW1 
READ(5,85)PX1 
READ( 5 ,85) PY1 
READ ( 5 , 85 ) PZ 1 

84  FORMAT  (46H  ENTER  A B 

85  FORMAT  (F8.2) 

START  DECIDE 

END 


C D COST1  PI1  PU  PV  PW1  PX1  PY1 


PZ1) 
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.RUN  CTREE(31 ,76) 


*★***★★**★★★★******★*★★★★*★★★★*★*★*★*★★★ 

* DECISION  TREE  EVALUATION  SYSTEM  * 

* PREPARED  FOR  ARPA  * 

* BY  THE  SRI  DECISION  ANALYSIS  GROUP  * 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★a**** 


TREE  LANGUAGE  FILE:  NAVAL 

FORTRAN  FILEiFNORT 
FORTRAN:  NF0RT.F4 

LOADING 

NFORT  8K  CORE 
EXECUTION 

TREE  FILE:  NTREE 

ENTER  A B C D COST1  PI1  PU  PV  PW1  PX1  PY1  PZ1 

28.0 

-3.8 

-.5 

0.0 

-11.1 

.95 

.75 

.50 

.85 

.90 

.75 

.50 


TREE  FILE:NTREE 
TREE  HAS 


11  NODES 
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COMMAND: RUN 


LOTTERY  , ACTIVE  NODE  PRINTOUTS,  STARTING  NODE,  OK  ?YES 


CERTAIN  EQUIV.  OF  STARTING  NODE:  9.3183998 

SELECTED  DECISION:  RKPKG  ALTERNATIVE  2 NODE  NO.  10 

ANALYSIS  OF  ALTERNATIVES 


NAME 

ALT. 

NODE  NO. 

CERTAIN  EQUIVALENT 

NORIS 

1 

3 

8.9500000 

RKPKG 

2 

10 

9.3183998 

SUCCESSOR 

NODE 

NODE 

DEC- 

NODE 

NODE 

ROLLBACK 

NO. 

NAME 

ISION 

NO. 

NAME 

VALUE 

11 

DECID 

2 

10 

RKPKG 

9.318400 

10 

RKPKG 

20.41840 

9 

PFAIL 

1 

8 

ACPT1 

8.950000 

8 

ACPT1 

9.950000 

7 

IMPL1 

12.10000 

6 

PSUCC 

1 

5 

ACPT1 

21 .02200 

5 

ACPT1 

21 .02200 

4 

IMPL1 

24.82000 
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LOTTERY  ON  ALL  REWARDS  IN  TREE. 


LOTTERY  MIN*  -14.900000 
MEAN*  9.3184000  SD= 


MAX*  16.900000 
13.001403 


LOTTERY 

VALUE 


PROBABILITY  CUMULATIVE 


-14.90000 

-11.60000 

16.90000 


0.09950 

0.15500 

0.74550 


0.09950 

0.25450 

1.00000 


1.0000- 

+ 

+ 

+ 

+ 

0.8000- 

+ 

+ 

+ 

+ 

0.6000- 

+ 

+ 

+ 

+ 

0.4000- 

+ 

+ 

+ 

+ 

0.2000- 

+ 

. + 
+ 
+ 

0.0000- 


0 


9 


-22.500  -15.000  -7.500 


• ,+++++++++  ,nmim  ,+++++++++  ,++ 
0.000  7.500  15.000  22.500 
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COMMAND:  GAMMA 


RISK  AVERSION  C0EF.=.02 


COMMAND: RUN 

GAMMA=  0.20000000E-01 , LOTTERY  , ACTIVE  NODE  PRINTOUTS,  STARTING  NODE,  OK  ?YES 

CERTAIN  EQUIV.  OF  STARTING  NODE:  7.4712394 

SELECTED  DECISION:  RKPKG  ALTERNATIVE  2 NODE  NO.  10 

ANALYSIS  OF  ALTERNATIVES 


NAME 

ALT. 

NODE  NO.  CERTAIN  EQUIVALENT 

NORIS 

1 

3 6.8869030 

RKPKG 

2 

10  7.4712394 

SUCCESSOR 

NODE 

NODE 

DEC- 

NODE 

NODE 

ROLLBACK 

NO. 

NAME 

ISION 

NO. 

NAME 

VALUE 

11 

DECID 

2 

10 

RKPKG 

7.471239 

10 

RKPKG 

18.57124 

9 

PFAIL 

1 

8 

ACPT1 

6.886903 

8 

ACPT1 

6.886903 

7 

IMPL1 

9.613395 

6 

PSUCC 

1 

5 

ACPT1 

19.26884 

5 

ACPT1 

19.26884 

4 

IMPL1 

28.74201 
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COMMAND: LIST 

FORWARD  OR  BACKWARD:  FORWARD 
NODE  DECID  NO.  11  TYPE  DEC 
ATTR:  1.0000000 

SIJfTFSSOR  3 

SUCCESSOR  10  ATTR:  -11.100000 

NODE  RKPKG  NO.  10  TYPE  CHA 
ATTR:  5.0000000 

SUCCESSOR  6 ATTR:  0.95000000 

SUCCESSOR  9 ATTR:  0.49999997E-01 

NODE  PFAIL  NO.  9 TYPE  DEC 

ATTR:  9.0000000 
ciirrcccnp  o 

SUCCESSOR  0 ATTR: -0.50000000 

NODE  ACPT1  NO.  8 TYPE  CHA 
ATTR:  10.000000 

SUCCESSOR  7 ATTR:  0.75000000 

SUCCESSOR  0 ATTR:  0.25000000 

NODE  IMPL1  NO.  7 TYPE  CHA 
ATTR:  11.000000 

SUCCESSOR  0 ATTR:  0.50000000 

SUCCESSOR  0 ATTR:  0.50000000 

NODE  PSUCC  NO.  6 TYPE  DEC 
ATTR:  6.0000000 

SUCCESSOR  5 

SUCCESSOR  0 ATTR:  0.0000000 

NODE  ACPT1  NO.  5 TYPE  CHA 
ATTR:  7.0000000 

SUCCESSOR  4 ATTR:  0.85000000 

SUCCESSOR  0 ATTR:  0.15000000 

NODE  IMPL1  4 TYPE  CHA 

ATTR:  8.0000000 

SUCCESSOR  0 ATTR:  0.90000000 

SUCCESSOR  0 ATTR:  0.10000000 

NODE  NORIS  NO.  3 TYPE  DEC 
ATTR:  2.0000000 

currcccno  ? 

SUCCESSOR  0 ATTR:  0.0000000 

NODE  ACPTO  NO.  2 TYPE  CHA 
ATTR:  3.0000000 

SUCCESSOR  1 ATTR:  0.75000000 

SUCCESSOR  0 ATTR:  0.250000000 

NODE  IMPLO  NO.  1 TYPE  CHA 
ATTR:  4.0000000 

SUCCESSOR  0 ATTR:  0.50000000 

SUCCESSOR  0 ATTR:  0.50000000 


-0.50000000 

28.000000 

-3.8000000 

-0.50000000 

28.000000 

-3.800000 

-0.50000000 

28.000000 

-3.8000000 
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COMMAND: GAMMA 
RISK  AVERSION  COEF.'O 


COMMAND: RUN 

LOTTERY  , ACTIVE  NODE  PRINTOUTS,  STARTING* NODE,  OK7YES 

CERTAIN  EQUIV.  OF  STARTING  NODE:  9.3193998 

SELECTED  DECISION:  RKPKG  ALTERNATIVE  2 NODE  NO.  10 


ANALYSIS  OF  ALTERNATIVES 


NAME 

ALT. 

NODE  NO. 

CERTAIN  EQUIVALENT 

NORIS 

1 

3 

8.9500000 

RKPKG 

2 

10 

9.3183998 

NODE 

NODE 

DEC- 

SUCCESSOR 
NODE  NODE 

ROLLBACK 

NO. 

NAME 

ISION 

NO.  NAME 

VALUE 

11 

DECID 

2 

10 

RKPKG 

9.318400 

10 

RKPKG 

20.41840 

9 

PFAIL 

1 

8 

ACPT1 

9.950000 

8 

ACPT1 

8.950000 

7 

IMPL1 

12.10000 

6 

PSUCC 

1 

5 

ACPT1 

21.02200 

5 

ACPT1 

21 .02200 

4 

IMPL1 

24.82000 
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SECTION  V 
SCENARIO  EXAMPLES 


This  section  contains  examples  of  applying  ETAM  to  all  or  part  of  the 
recommended  process  for  evaluating  a training  innovation.  The  first  scenario 
takes  a proposed  innovation  from  the  innovator's  description  through  the 
series  of  ETAM  tasks.  The  results  of  each  task  are  presented  as  a general, 
but  not  complete,  model  for  the  analytic  and  decision  process.  The  remaining 
scenarios  are  treatments  of  subsets  of  the  assessment  process  which  were 
felt  to  require  further  elaboration.  Also,  some  sample  innovations  have 
been  included  to  which  ETAM  procedures  might  be  applied. 

SCENARIO  NUMBER  ONE  - 3-0  PROCEDURAL  TRAINER 

The  innovator  experimented  with  a 3-D  simulator  for  replacing  actual  opera- 
tional laboratory  training  equipment  used  in  certain  training  tasks.  The 
innovator's  description  of  his  preliminary  work  is  presented  as  the  starting 
point  for  the  scenario.  The  reviewer  should  be  aware  of  the  following  as- 
pects of  the  approach  to  presenting  this  scenario  material. 

1.  Inconsistencies  in  the  Task  1 description  are  deliberately  introduced 
to  simulate  realisms.  Actual  people  don't  always  see  the  real  versus 
fancied  benefit  or  liability  of  the  product,  even  when  the  data  stare 
them  in  the  face.  This  is  one  of  the  values  of  formal  ETAM  which  sheds 
light  on  all  of  it. 

2.  The  present  scenario  is  an  abstract  of  what  might,  in  real  life,  be  a 
lot  of  text.  But  it  should  be  evident  that  text,  other  than  the  context 
of  what  appears  here,  would  be  superfluous  and  even  obscuring. 

3.  The  major  kind  of  data  that  might  have  been  added  would  have  been  his- 
torical field  experiences  showing  how  poorly  most  people  do  on  proce- 
dures, and  the  high  error  rates  almost  universally  associated  with 
procedural  tasks. 

INNOVATOR'S  DESCRIPTION.  The  innovator  prepared  the  following  write-up  as 
an  input  to  the  analysis  and  decision  process. 

EXHIBIT:  3-D  SIMULATION  OF  EQUIPMENT  TO  TEACH  EQUIPMENT  PROCEDURES 

Descripti on 

On-the-job  evaluation  of  operators  and  maintenance  personnel  shows  a 
general  inadequacy  in  the  performance  of  routinized  procedures  such 
as  turn  on,  calibration,  checkout,  operation,  and  so  forth.  This 
leads  to  inefficient  performance  and  many  errors. 

But  the  use  of  actual  equipment  in  training  exercises  is  expensive  in 
many  ways.  The  cost  of  the  equipment,  plus  its  maintenance  and  cali- 
bration for  training,  limits  the  number  which  is  practical  for  training 
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EXHIBIT:  3-D  SIMULATION  OF  EQUIPMENT  TO  TEACH  EQUIPMENT  PROCEDURES 

(Continued) . . . 


purposes.  Furthermore,  an  instructor  must  usually  be  present  for  each 
one  or  pair  of  students  working  on  equipment  so  that  the  correctness 
of  each  step  can  be  verified,  or  to  provide  guidance  and  interpretation 
when  things  go  wrong,  either  because  of  student  error  or  equipment  fail- 
ure. The  combination  of  these  costs  may  limit  student  practice  on  the 
equivalent  of  hands-on  procedure  to  a few  hours,  rather  than  the  dozens 
of  hours  necessary  for  a reliable  proficiency. 

This  innovation  consists  of  using  a dummy  shell  of  the  equipment,  but 
with  all  indicators  and  controls  having  an  appearance  of  being  opera- 
tional. Controls  are  manipulable.  External  test  points  on  which  to 
attach  test  equipment  probes  are  presented  and  "functional." 

The  internals  of  the  mockup  consist  of  circuits  which  feed  impulses  to  a 
programmed  computer.  The  latter  returns  equipment  indications  suitable 
to  the  control  activations.  In  addition,  the  program  generates  correc- 
tive feedback  to  the  student.  This  is  presented  on  a nearby  screen. 

The  screen  can  present  only  static  images,  but  it  can  display  a given 
wave  shape  in  response  to  a given  situation  and  action. 

Several  comparative  studies  had  the  following  results: 

1.  The  experimental  group  was  given  the  same  overall  number  of  lab 
sessions  and  lab  hours  as  the  control  group  — the  use  of  live 
equipment  was  by  the  control  group.  In  the  experimental  group, 
each  student  had  a teaching  mockup  which  he  used  throughout  the 
lab  periods,  except  for  the  final  performance  test  which  was  made 
on  two  pieces  of  live  equipment  and  monitored  by  the  instructor. 

The  experimental  group's  performance  on  a scale  which  combined 
scores  on  speed  and  errors,  with  a high  weighting  on  errors,  was 
judged  to  be  40%  superior  to  the  performance  of  the  control  group. 

Retest  a month  later,  showed  the  experimental  group  to  be  60% 
superior  to  the  control  group.  These  scores  suggested  an  estimated 
reduction  in  on-the-job  training  from  four  weeks  to  one  week  in 
order  to  acquire  acceptable  operational  efficiency. 

Tests  six  months  after  incumbents  were  on  the  job  showed  30%  of 
the  experimental  group  as  "outstanding"  in  performance  on  the 
subject  tasks,  whereas  only  5%  of  the  control  group  was  judged 
outstanding  In  their  job  performance.  There  were  similar  superi- 
orities of  the  experimental  over  the  control  subjects  at  other 
levels  of  evaluation. 

During  training,  there  was  1%  attrition  in  the  experimental  group 
and  7%  in  the  control  group. 
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EXHIBIT:  3-D  SIMULATION  OF  EQUIPMENT  TO  TEACH  EQUIPMENT  PROCEDURES 

(Continued) ... 


2.  Cost  analysis.  Development  plus  operational  costs  of  the  ex- 
perimental device  netted  out  to  $10  per  student  hour.  The  costs 
for  the  actual  hardware  (control  condition)  netted  out  to  $9  per 
student  hour.  These  costs  were  based  on  a 5-year  spread. 

(One  liability  in  the  experimental  device,  the  operating  mockup, 
was  the  need  to  run  one  pass  of  pilot  students  through  the  device 
in  actual  training  conditions  in  order  to  properly  debug  its  opera- 
tion. For  this  class,  the  ordinary  complement  of  actual  equip- 
ments had  to  be  available  as  backup  for  training  while  the  mockup 
trainers  were  being  debugged..) 

In  the  control  group,  one  instructor  was  necessary  for  every  four 
students,  while  In  the  experimental  group,  one  instructor  was 
sufficient  for  a class  of  20  to  25  students. 

Problem 


A large  Investment  would  have  to  be  made  by  the  Navy  to  develop  the 
technology  for  the  mockup  trainer.  It  could  be  connected  by  telephone 
to  a central  computer.  The  ultimate  costs  for  units  of  this  device 
could  be  drastically  reduced  if  they  could  be  used  in  very  large  num- 
bers in  a large  range  of  job  tasks.  (The  cost  analysis  figure  for  the 
device,  cited  above  as  $10,  assumed  a minimum  of  a thousand  of  the  de- 
vices In  use  by  the  Navy  at  the  end  of  5 years.  A large  investment  in 
development  funds  might  have  a good  chance  of  reducing  the  cost  per 
student  hour  below  the  $9  stated  above.) 

It  Is,  therefore,  Important  to  determine  the  widest  potential  range- 
of-effect  for  this  training  function. 

TASK  1 - FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION.  The  innovator  ex- 
pands his  description  of  the  innovation  by  referring  to  the  Task  1 proce- 
dures. The  assessor-consultant  may  participate  in  this  phase,  however, 
this  is  not  a necessity.  A sample  Descriptive  Data  Sheet  prepared  by  the 
Innovator  might  appear  as  follows.  Task  and  subtask  references  are  identi- 
fied where  appropriate. 

EXHIBIT:  DESCRIPTIVE  DATA  SHEET 

NAME  OF  THE  INNOVATION 
3-0  Procedural  Trainer 

OBJECTIVES  AND  SUMMARY  DESCRIPTION  (01.01) 

The  objective  is  to  Increase  the  efficiency  and  retentivlty  of  the 
learning  of  operating  procedures  on  electronic  equipment.  The  device 
consists  of  the  shell  of  the  electronic  gear  on  which  the  procedures 
are  to  be  learned.  Within  this  shell,  special  circuits  are  hooked  up 
to  a computer  which  simulates  the  behavior  of  the  equipment  to  the 
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EXHIBIT:  DESCRIPTIVE  DATA  SHEET  (Continued)... 


actions  of  the  student,  but  also  provides  corrective  and  other  in- 
structional guidance.  The  computer  is  programmable  so  as  to  adapt 
to  the  learning  objectives. 

DELIVERY  SYSTEM  (01.02) 

The  innovation  centers  around  a delivery  system  for  procedures  learn- 
ing. The  trainer  displaces  the  operational  live  equipment  that  is  used 
for  instructional  purposes  and  the  instructional  manuals  that  guide  the 
student  in  his  learning  the  procedure. 

Instructional  Process 


The  device  enables  the  student  to  perform  manual  procedures  with  actual 
displays  and  controls  and  their  relationships.  He  also  gets  instruc- 
tional feedback  and  guidance  from  an  auxiliary  display  screen  controlled 
by  the  computer.  The  student  actually  learns  by  doing  the  task,  with 
supplemental  Individualized  help  and  direction  controlled  by  the  re- 
sponses he  has  already  made,  and  should  be  making  next.  This  should 
result  In  faster  learning,  better  remembering,  and  higher  procedural 
level  skill.  The  device  is  a form  of  simulator. 

JOBS  AND  TASKS  (01.03) 

Of  primary  relevance  are  those  jobs  that  deal  with  the  external  displays 
and  controls  of  intact  electronic  equipment.  This  may  include  test 
equipment  and  operator  training  in  sequential  procedures  on  the  equip- 
ment such  as  turn  on,  start  up,  checkout,  and  operation.  Also,  mainte- 
nance tasks  done  on  the  outside  of  the  intact  electronic  equipment  such 
as  checkout,  alignment,  and  so  forth. 

Equipment  Orientation 

Electronic  equipment.  No  restrictions  on  equipment  type. 

Work  Function 


Sequential  procedures:  reading  indicators;  activating  controls  as  pre- 

scribed in  sequence;  recognizing  deviations  from  "normal"  and  taking 
specified  action. 

Operational  Environment 

Any  lab  environment:  no  physical  motion;  normal  temperatures.  The 

student  is  unstressed  except  as  may  be  artifically  induced  by  verbal 
means.  Space  used  by  the  trainer  is  practically  equivalent  to  that 
used  by  operational  equipment  for  training. 
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EXHIBIT:  DESCRIPTIVE  DATA  SHEET  (Continued)... 

STUDENT  ATTRIBUTES  (01.04) 

May  help  slow  learners  of  routine  tasks.  Student  attribute  factors 
are  not  a major  target  of  the  innovation. 

Prior  Training  and  Experience 

The  trainer  could  be  an  efficient  way  of  providing  refresher  training, 
but  this  has  not  been  investigated  as  a justification. 

Attitudes  and  Motivations 


These  should  be  improved  as  a by-product  of  the  student  getting  im- 
personal, individual,  and  continuous  practice  rather  than  waiting  around 
for  the  instructor's  attention,  or  making  mistakes  and  not  knowing  what 
he  did  incorrectly  for  substantial  periods  of  time.  The  student  may, 
early  in  learning  the  task,  get  a proper  sense  of  pace  which  can  be 
forced  by  the  programming  of  the  instructional  guidance. 

Aptitudes 


These  may  be  affected,  but  only  as  a by-product  of  other  factors.  The 
trainer  can  reduce  the  dependence  of  instructor  and  student  on  the  stu- 
dent's ability  to  learn  symbolic  materials  which  may  not  be  highly  cor- 
related with  his  ability  to  learn  the  procedures  by  actual  physical 
operations. 

Social  Factors 


No  intended  influence. 
Setbacks  and  Attrition 


The  reduction  of  these  is  a secondary  target. 
Student  Evaluation 


A continuous,  objective  record  of  the  errors  made  by  the  student  is  main- 
tained. This  should  be  superior  to  instructor  evaluations.  The  device 
may  be  used  to  keep  the  student  practicing  until  the  desired  performance 
criterion  is  reached. 

EMPIRICAL  DATA  (01.05) 

Study  performed  on  two  pieces  of  equipment. 
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EXHIBIT:  DESCRIPTIVE  DATA  SHEET  (Continued)... 

Quantitative  Benefits 
Measured  Results 

Control:  Normal  training  operations. 

Exper:  Use  of  innovation  on  same  equipments,  same  procedures 

to  be  learned. 

Results:  Tested  on  live  equipment  after  equal  hours  of  elapsed 

time. 

1.  Immediately  after  training:  Exper  were  40%  superior. 

2.  Retest  one  month  later:  Exper  60%  superior.  (Equivalent 

to  reducing  OJT  from  4 weeks  to  1 week.) 

3.  Test  six  months  after  on  job: 

Exper:  30%  rated  "outstanding'1' on  job. 

Contr:  5%  rated  as  "outstanding." 

4.  Attrition 

Exper:  1% 

Contr:  7% 

Criterion  Variables  for  Measuring  Training  Effectiveness 

Validity  and  completeness  in  level  of  proficiency. 

Target:  Higher  level  of  operational  proficiency  in 
dures  performance. 

Quantitative  Costs 

Estimated  operation  costs  for  innovation,  8 years:  $7.00  per 

hour. 

Present  operating  cost  for  training:  $8.50  per 

hour. 

Exper:  One  Instructor  per  20  students. 

Contr:  One  Instructor  per  4 students. 


proce- 

student 

student 


V-6 


TAEG  RE PORI  NO.  12-3 


EXHIBIT:  DESCRIPTIVE  DATA  SHEET  (Continued)... 


Estimates  of  Magnitudes  of  Beneflts/Savinqs  under  Actual  Conditions 

80%  reduction  in  attrition  rate. 

75%  savings  in  OJT  per  Incumbent. 

Reduction  In  standby  and  spare  equipment. 

Decreases  In  mission  failures. 

ESTIMATES  OF  THE  PRACTICAL  IMPORTANCE  OF  THE  TRAINING  PRODUCT  (01.06) 

Practical  procedural  proficiency:  crucial  Importance  to  operations. 

RESOURCE  REQUIREMENTS  FOR  FURTHER  STUDY  (01.07) 

Technical  development:  $12  million  over  3 years. 

Empirical  studies:  $ 3 million  over  3 years. 

TASK  2 - DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION.  The  development  of 
alternatives  may  require  extensive  study,  reference  to  the  literature,  and 
consultation  with  experts.  In  this  scenario,  the  assessment  should  be  per- 
formed in  two  steps.  First,  alternatives  generated  should  be  considered 
against  the  same  objectives  as  the  original  proposal  (3-D  Trainer).  Second, 
any  additional  benefits  specific  to  the  alternative  should  be  Identified  and 
assessed.  The  following  briefly  describes  an  alternative  to  the  3-D  Trainer. 
If  this  should  become  an  important  consideration  in  the  decision  process,  it 
will  be  subjected  to  the  same  descriptive  procedure  (Task  1)  and  analysis  as 
the  original  proposal. 

EXHIBIT:  INNOVATION  ALTERNATIVES 

CMI  Display  Terminals 

Microfiche  frames,  random  selection  by  computer. 

Get  high  mediated-performance  proficiency. 

Then  final  training  on  live  equipment. 

Major  Advantage 

All  technology  in  place. 

Attrition  reduction  from  7%  to  5%. 

Student/Instructor  ratio  Increased  from  4 to  5. 
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EXHIBIT:  INNOVATION  ALTERNATIVES  (Continued)... 

Major  Liability 

Slight  cost  increase  ($4,375.00  to  $5,000.00). 

TASK  3 - MAKE  PRELIMINARY  FEASIBILITY  PROFILE.  The  following  risk  statements 
were  selected  from  the  list  appearing  In  each  of  the  major  potential  risk 
categories.  They  are  applicable  to  the  3-D  Procedural  Trainer. 

EXHIBIT:  PRELIMINARY  FEASIBILITY  PROFILE  (Statements  of  Risk) 

IMPORTANCE  TO  NAVY  MISSION  (03.01) 

Of  moderate  Importance. 

ORGANIZATIONAL  COMPATIBILITY  (03.02) 

Irrelevant  to  Navy  comnand  and  career  structures. 

GOALS/POLICY  COMPATIBILITY  (03.03) 

Support  of  policy  to  Increase  practical  skill  levels. 

STATE  OF  THE  ART  (03.04) 

Moderate  probability  of  success  with  some  continued  R&D  effort. 

Problems: 

1.  Hardware  linkages  of  equipment  to  computer  sensors. 

2.  Simulation  of  temporal  response  to  operation  actions. 

3.  Reliability  of  hardware/software  performance. 

4.  Lead  time  needed  to  program  the  training  functions  on  device. 
FUNDING  (03.05) 

Large  or  very  large  Investment  to  bring  the  innovation  to  an  operational 
level  of  implementation  (near  maximum  that  can  be  conmltted  to  one 
project). 

TECHNICAL  SUPPORT  (03.06) 

Some  difficulty  In  getting  R&D  talent  and  Implementation  talent. 
ATTITUDINAL  ACCEPTANCE  (03.07) 

Moderate  but  temporary  resistance  by  instructional  community,  especially 
If  there  will  be  difficulties  In  operation,  equipment  reliability,  or 
system  availability  and  responsiveness. 
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EXHIBIT:  INNOVATION  ALTERNATIVES  (Continued)... 


Student  acceptance. 

PRELIMINARY  FEASIBILITY  PROFILE  SUMMARY  (03.08) 

Figure  V-l  summarizes  the  analysis  from  03.01  through  03.07,  showing 
the  innovation  as  moderately  risky  with  a number  of  risk  reduction 
projects  and  associated  dollars  required  to  bring  the  implementation 
and  use  of  the  innovation  to  an  acceptable  level  of  risk. 

Task  4 - PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT.  The  preliminary  assess- 
ment of  risks  performed  in  Task  3 is  refined  at  this  point;  success  pro- 
babilities are  determined,  projects  are  costed,  and  a decision  variable 
is  calculated  based  upon  the  tangible  factors.  The  following  are  examples 
of  results  from  Task  4 procedures  applied  to  the  analysis  of  the  3-D 
Procedural  Trainer. 

EXHIBIT:  ANALYTIC  FEASIBILITY  ASSESSMENT 

POTENTIALLY  HIGH  RISK  AREAS  (04.01) 

State  of  the  Art  (SA) 

SA1 . Training  laboratory  locations  will  present  some  problems 
in  tying  student  stations  to  a remote  computer  facility. 
Present  technology  permits  line  lengths  of  up  to  2000 
feet  to  be  driven  at  reasonable  cost.  Some  facilities 
are  a half-mile  from  the  computer  site.  R&D  efforts 
should  be  devoted  to  finding  a technology  capable  of 
driving  lines  up  to  one  mile  at  reasonable  cost. 

SA2.  Present  computational  speeds  would  cause  delays  of  up  to 
4 seconds  in  responding  to  operation  actions.  This  may 
affect  student  acceptance  of  the  device,  but  more  so, 
may  impair  the  learning  of  certain  operational  procedures 
where  a sequence  of  actions  must  be  learned  and  where 
more  rapid  feedback  may  be  essential.  An  assessment 
(study)  of  the  approximate  amount  of  training  taking 
place  on  actual  equipment,  which  requires  faster  response 
that  is  now  possible,  should  be  made.  Additional  study 
should  also  be  directed  at  determining  student  attitude 
effects,  especially  as  they  might  relate  to  decreased 
learning  effectiveness. 

SA3.  Availability  of  student  trainers  will  be  the  key  to  achiev- 
ing desired  objectives.  Present  computer  reliability  would 
appear  to  indicate  an  availability  of  between  90%  and  95%. 

It  must  be  determined  if  sufficient  training  contingency 
plans  can  be  adopted  to  cope  with  this  level  of  reliability. 

SA4.  The  proposed  trainer  is  general  purpose  and  its  uniqueness 
to  the  training  situation  will  primarily  result  from 
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MODERATE 
IMPORTANCE 

NO  RISK 
OR  OF 

SUBSTANTIAL 

IMPORTANCE 

SUPPORTIVE 
OR  OF  HIGH 
OR  CRUCIAL 
IMPORTANCE 

COMMENTS, 

NATURE  OF  RISK,  ETC. 

NUMBER  OF  RISK 
REDUCTION  PRO- 
JECTS REQ.'D 
AND  IDENTIFIED 

ESTIMATED 

PROJECT 

COSTS 

IMPORTANCE  TO 
PROJECTED  NAVY 
MISSION 

XX 

X 

X 

ORGANIZATIONAL 

COMPATIBILITY 

XX 

0 

0 

» 

GOALS/POLICY 

COMPATIBILITY 

XX 

0 

0 

STATE-OF-THE-ART 

XX 

HARDWARE  LINKAGES  & 
REL.  TEMP.  RESP.  SIMU- 
LATION PROG. LEAD 
TIME 

4 

1 1,000K 

R&D  FUNDING 
REQUIREMENTS 

XX 

NEAR  MAXIMUM 
FOR  SINGLE 
PROJECT 

0 

0 

TECHNICAL  SUPPORT 
REQUIREMENT 

XX 

SOME  DIFFICULTY 
OBTAINING  R&D  IMPLE- 
MENTATION TALENT 

0 

0 

ATTITUDINAL 

ACCEPTANCE 

XX 

TEMPORARY  RESIST- 
ANCE BY  INSTRUCTIONAL 
COMMUNITY 

1 

55K 

OVERALL  PROFILE 
AND  TOTAL  PROJECTS 
AND  DOLLARS 

XX 

5 

11,100K 

FIGURE  V-1.  SUMMARIZED  PRELIMINARY  FEASIBILITY  PROFILE 
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EXHIBIT:  ANALYTIC  FEASIBILITY  ASSESSMENT  (Continued)... 


programming  the  Interrelationships  between  operator  con- 
trols and  feedback  signals  (lights,  meters,  etc.).  Pro- 
gramming time  was  assessed  for  the  experimental  applica- 
tion, but  no  specific  techniques  (e.g.,  high  level 
language,  subroutine  libraries,  etc.)  were  analyzed  for 
speeding  the  preparation  of  programs.  A major  study 
should  be  undertaken  to  investigate  possible  techniques 
for  improving  programming  efficiency. 

Attitudinal  Acceptance  (AA) 

AA1.  Acceptance  of  the  instructional  community  will,  to  a 
great  extent,  be  a function  of  successfully  overcoming 
the  risk  items  stated  above.  Excessive  system  downtime 
or  slow  response  time  could  seriously  impact  student 
acceptance.  Appropriate  orienting  presentations 
should  be  prepared  to  help  overcome  resistance  of 
administrators  and  students. 

SUCCESS  PROBABILITIES  (04.02) 

Implementation  Risks  75%  (65%  to  85%) 

SA1  - Hardware  line  linkages. 

SA3  - Hardware/software  reliability. 

SA4  - Programing  time. 

User  Acceptance  Risks  50%  (40%  to  60%) 

SA2  - Response  delays. 

SA3  - Hardware/software  reliability. 

AA1  - Improper  user  orientation. 

IMPLEMENTATION  PROJECTS  (04.03) 

Project  Name 


User  Orientation  for  3-D  Trainer  (AA1) 
Abstract 


Implementation  of  the  3-D  Procedural  Trainer  throughout  the  training 
command  will  be  accompanied  by  a great  deal  of  resistance,  especially 
at  the  command  levels.  One  solution  to  this  problem  is  to  ensure 
that  an  adequate  understanding  of  the  device  exists  with  officers 
and  staff  who  will  be  responsible  to  manage  the  implementation  and 
operation  of  these  devices  within  their  command.  This  will  be 
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EXHIBIT:  ANALYTIC  FEASIBILITY  ASSESSMENT  (Continued)... 

accomplished  by  developing  a 15-minute  color  movie  on  the  device,  its 
features,  and  Its  application,  for  presentation  to  command  level 
officers  and  staff  personnel  prior  to  the  planning  stage  for  one  of 
these  devices.  Three  persons  will  require  a total  of  12  man-months 
to  develop  the  movie  at  an  estimated  cost  of  $54,215.  Development 
will  be  scheduled  to  begin  May  1976  and  complete  April  1977. 

Objectives 

Develop  15-minute  color  movie  on  3-D  Procedural  Trainer. 

Improve  acceptance  of  users  at  command  level. 

Project  Plan  Milestones 


Prepare  final  Implementation  plan. 

Jan 

1976 

Obtain  funding  approval. 

Mar 

1976 

Recruit  personnel. 

Apr 

1976 

Begin  design-consultation  trips. 

May 

1976 

Complete  design. 

Oct 

1976 

Begin  photography. 

Nov 

1976 

Complete  photography. 

Feb 

1977 

Obtain  prints. 

Apr 

1977 

Distribute  prints  for  use. 

May 

1977 

Preliminary  Cost  Estimate 
(See  Figure  V-2.) 

NOTE:  The  project  description  will  be  as  detailed  as 

required  for  documenting  the  project  need. 

REEVALUATE  SUCCESS  PROBABILITIES  (04.04) 

With  successful  accomplishment  of  the  risk  reduction  projects.  It  is 
anticipated  that  success  probabilities  will  be: 

Implementation  85$  (80*  to  90*). 

User  Acceptance  90*  (85*  to  95*). 

Project  success  Is  estimated  at  95*  (90*  to  100*).  If  the  project 
package  falls,  probabilities  estimated  In  04.02  will  prevail. 
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PRELIMINARY 
COST  ESTIMATE 

NEW  E2J  REVISEDC 


PROJECT 


PROJECT  NAME 

USER  ORIENTATION  FOR  3-D  TRAINER 


PROJEQT  NO. 

63941  AA-1 


DATE 

30  JULY  1975 


PERSONNEL 


JOB 


MOVIE  DESIGN  LAYOUT  (GS-12) 


PHOTOGRAPHY  (PH6) 


ADMINISTRATIVE 


MAN  MONTHS 


COST/MAN  MONTH 


1855.00 


1630.00 


970.00 


JOB  COST 


11130.00 


6520.00 


1940.00 


$ 19590.00 


EQUIPMENT 


DESCRIPTION 


PROTOTYPE  DEVICE  FOR  USE  IN 


DESIGN  OF  SEQUENCES  AND 


FILMING 


QUANTITY 


UNIT  COST 


12400.00 


COST 


12400.00 


$12400.00 


SPACE 


REASON 


SQUARE  FEET 


COST/SQ,  FT. 


COST 


TRAVEL  (NON-STUDENT! 


FROM  - TO  REASON 


MULTI-LOCATION  TRAVEL  FOR 


CONSULTATION  AND  FILMING 


NO.  OF  TRIPS 


20 


COST/TRIP 


280.00 


COST 


5600.00 


S 5600.00 


STUDENTS 


NO. OF  STUDENTS  - REASON 


NO.  OF  DAYS 


COST/STUDENT  DAY  COST 


OTHER 


DESCRIPTION 


PROCESSING  AND  REPRODUCTION 


OF  100  COPIES 


MAN  MONTHS/QTY. 


100 


UNIT  COST 


5625.00 


110.00 


COST 


5625.00 


11000.00 


S 16625.00 


TOTAL  PROJECT  COSTS 


S 54215.00 


FIGURE  V-2.  RISK  REDUCTION  PROJECT  COST  ESTIMATE 
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EXHIBIT:  ANALYTIC  FEASIBILITY  ASSESSMENT  (Continued)... 

PRELIMINARY  RANGE-OF-EFFECT  (04.05) 

Preliminary  Range  of  Courses 

Based  upon  the  experimental  results  in  course  XXXX,  a preliminary  re 
view  of  similar  type  courses  shows  the  following  potential  for  appli 
cation  of  the  3-D  Procedural  Trainer. 


Course 

Course  Length 

Average  Annual  AOB 

XXXX 

15  days 

6500 

YYY 

% 

9 days 

1300 

(etc.) 

• 

Major  Cost  Categories  (Projected  Differences) 

Student  Salaries 

+$  480K  (Savings) 

Student  Travel 

+$  360K  (Savings) 

Instructor  Salary 

+$  4,375K  (Savings) 

Equipment 

-$  537K  (Cost) 

On-Job-Training 

+$23,364K  (Savings) 

Net  Savings 

$28,042K 

NOTE:  The  above 

savings  assume  no 

risk  reduction 

expenditures  and  that  implementation  is 
successful  and  there  is  user  acceptance,  so 
that  the  full  eight  years  of  savings  are 
realized.  This  cost  will  be  assigned  to 
Decision  Tree  Outcome  A.  (See  Figure  V-3.) 

Outcome  B (Assumptions) 

Two-year  cycle  for  measuring  user  acceptance. 

Existing  plan  will  be  implemented  for  final  six  years  of  planning 

period. 

• Original  equipment  will  be  purchased. 

• 3-D  Trainers  will  be  scrapped  (no  salvage  value). 

Anticipated  Cost  - $3,800K. 
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EXHIBIT:  ANALYTIC  FEASIBILITY  ASSESSMENT  (Continued)... 


NOTE:  Similar  analyses  would  be  performed  for 

each  outcome. 

The  savings  and  costs,  as  well  as  the  estimated  success  probabilities, 
are  placed  on  the  decision  tree  diagram  and  folded  back.  The  prelim- 
inary decision  indicated  would  be  to  accept  the  proposal,  assuming 
that  the  risk  reduction  projects  involving  continued  study  would  be 
undertaken,  and  their  costs  considered  in  subsequent  analyses.  The 
decision  tree  is  shown  in  Figure  V-3. 

TASK  5 - DETERMINE  RANGE-OF-EFFECT.  Feasibility  has  been  examined  over  a 
limited  range  of  potential  application.  The  total  range-of-effect  for  the 
3-D  Procedural  Trainer  Is  determined  through  the  analytic  steps  of  Task  5. 

EXHIBIT:  RANGE-OF-EFFECT  ANALYSIS 

CHOOSE  MAJOR  ROUTE  FOR  DETERMINING  RANGE-OF-EFFECT  (05.01) 

Potential  Displaceable  Cost  in  Instructional  Vehicles  (Route  05.02) 
Instructors. 

Live  equipment  used  for  training  purposes. 

Procedure  trainers. 

Potential  Outcome  of  Training  in  a Skill  (Route  05.09) 

Search  strategy: 

Determine  relevant  courses. 

Derive  identification  of  instructors  and  instructional  devices. 
DEVELOP  BACKGROUND  FOR  TASK  CONTENT  TRAINING  ANALYSIS  (05.09) 

IDENTIFY  THE  LEARNING  OBJECTIVE  (05.10) 


See  Innovation  description: 

Practical  skill  training  in  routinized  procedures  applied  exter- 
nally to  electronic  equipment  and  maintenance. 

MAKE  PRELIMINARY  EVALUATION  OF  BENEFIT  IMPORTANCE  (05.11) 

Target  training  objective  is  extremely  important. 

Magnitudes  of  potential  improvement  are  very  substantial. 
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IMPORTANT  OUTCOMES: 


FIGURE  V-3.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
(3-D  PROCEDURAL  TRAINER  ANALYZED  OVER 
PRELIMINARY  RANGE-OF-EFFECT) 
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EXHIBIT:  RAN6E-0F-EFFECT  ANALYSIS  (Continued)... 

DETERMINE  IF  BENEFIT  APPLIES  TO  SKILL  OR  KNOWLEDGE  TRAINING  (05.12) 
This  is  practical  skill  training,  so  follow  Route  05.13. 

Type  of  skill:  Electronic  equipment,  procedural. 

SELECT  TASK  STRUCTURE  ELEMENT  OR  TASK  11  FUNCTION"  (05.13) 

Include  skill  practice  in  procedural  context  of: 

Goal -image 
Scan-detect 
Identify 
Interpret 

Manipulation  of  equipment  controls. 

Do  not  search  on  these  elements;  do  search  on  procedures,  operating 
and  maintenance,  on  electronic  equipment. 

IDENTIFY  RELEVANT  JOBS/TASKS:  CONTINUED  SEARCH  SPECIFICATIONS  (05.14) 

Job  Class:  Electronic  equipment  operations;  maintenance. 

Task  Characteristics: 

Equipment:  Electronic,  having  manual  procedures  associated  with 

it. 

Tools:  Test  equipment  for  electronic  equipment  servicing 

and  checkout. 

Reference  Information:  Procedures,  manuals  of  instruction. 

Examples  of  Task  Titles: 

Start-up  procedure 
Checkout  procedure 
Operating  procedure 

Alignment  procedure,  adjustment  procedure 
Diagnostic  or  troubleshooting  procedure,  external . 


V- 1 7 


TAEG  REPORT  NO.  12-3 


EXHIBIT:  RANGE-OF-EFFECT  ANALYSIS  (Continued)... 

SELECT  RELEVANT  STAGE  OF  LEARNING:  CONTINUED  SEARCH  SPECIFICATIONS 

(05.15) 

Not  orientation  or  familiarization  or  theory  of  operation.* 

Not  nomenclature  and  identifications.* 

Mediated  procedures. 

Unusual  task  conditions  (partial). 

Automated  performance  (partial). 

IDENTIFY  COURSE  RELEVANT  TO  INNOVATION  (05.16) 

AN/SRN-12  Omega  Receiving  Set  Maintenance  course. 

Total  training  time:  2 weeks. 

Practical  training  (lab  work):  6 days. 

IDENTIFY  RELEVANT  TRAINING  UNIT(S)  WITHIN  COURSE  (05.17) 

AN/SRN-12  Omega  Receiving  Set  Maintenance  course. 

Relevant  objectives  in  training: 

Receiving  set  operation  procedures. 

Lane  count  determination  procedures. 

Preventive  maintenance  procedures  using  test  equipment. 
Not  relevant:  troubleshooting  inside  covers  of  equipment. 

Estimated  training  time  for  all  practical  work  on  equipment:  6 days. 
Estimated  training  time  for  relevant  practical  work:  3 days. 

ASSESS  THE  BENEFIT  PATTERN  (05.18) 

AN/SRN-12  Omega  Receiving  Set  Maintenance  course. 

Relevant  training  target  time:  3 days. 

Relevant  equipment:  AN/SRN-12  Omega  Receiving  Set 


*Not  cost-effective  on  this  device. 
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EXHIBIT:  RANGE-OF-EFFECT  ANALYSIS  (Continued)... 


Training  Benefits 

a.  Estimated  reduction  In  attrition  rate:  80%  reduction  in 

rate  (from  7%  to  1%).  The  dollars  saved  by  the  attrition 
reduction  have  been  assigned  an  importance  rating  of  10 
due  to  present  funding  pressure. 

Confidence  range  — best  case,  85%  reduction  — worst  case, 
50%  reduction. 

b.  Liability  to  reduction  In  attrition  rate:  Inappropriate 

emphasis  on  evaluating  student  on  "theory  of  operation" 
content  in  course.  Would  not  affect  attrition  rate  on 
targeted  objective,  but  on  the  course  itself. 

NOTE:  If  the  reduction  In  attrition  had  value 

other  than  in  the  dollars  saved,  then  it 
would  be  assigned  an  Importance  level  as 
other  nonquantiflable  variables  are.  This 
might  be  the  case  where  the  particular 
manpower  pool  Is  in  short  supply. 

c.  Reduction  In  aptitude  requirements:  Potential,  but  not 

quantified. 

d.  Increased  flexibility  in  training  program  development. 
Importance  (rating  of  2). 

ON-THE-JOB  BENEFITS  (05.25) 

a.  Estimate  of  normal  OJT  before  acceptable  performance:  4 weeks. 

Estimate  of  OJT  with  innovation:  1 week. 

The  dollars  saved  by  the  OJT  reduction,  as  with  attrition 
dollars,  have  been  assigned  an  importance  rating  of  10. 

Confidence  range  --  best  case,  .5  week  with  innovation  — 
worst  case,  1.5  weeks. 

NOTE:  As  with  attrition,  if  increased  availability 

of  time  had  a value  independent  of  dollars. 

It  would  be  treated  as  a separate  variable. 

b.  Reduction  in  units  of  standby  equipment?  Estimated  30%  reduction. 

Confidence  range  --  best  case,  50%  reduction  --  worst  case, 

10%  reduction. 

Importance  (rating  of  1). 
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TASK  6 - PERFORM  COST-BENEFITS  ANALYSIS.  The  analysis  performed  in  Task  4 
will  be  refined  to  use  more  precise  cost  data,  to  assess  these  costs  over 
a broader  and  more  exacting  range-of-effect,  and  to  include  In  the  analy- 
sis some  of  the  less  tangible  benefits  previously  Identified. 

EXHIBIT:  COST-BENEFITS  ANALYSIS 

ASSEMBLE  DATA  (06.01) 

Obtain  Items  From  Prior  Tasks 


Decision  tree  from  completion  of  Task  4. 

List  of  courses  and  jobs  from  Task  5 to  which  Innovation  Is  expected 
to  apply. 

Benefits  pattern  to  be  applied  to  courses  and  jobs.  Including  magni- 
tude change  (amount  of  training;  l.e.,  course  length  and  number  of 
graduates  to  which  the  Innovation  Is  considered  applicable,  etc.)  and 
confidence  limits. 

Project  Descriptive  Data  and  Project  File  from  Tasks  1,  4,  and  5. 
Derive  Expected  Values  of  Variables 


The  expected  values  for  the  cost  models  and  decision  tree  input  vari- 
ables should  be  calculated  and  tabulated  In  a form  as  follows: 

Most  Standard 

Relevant  Variable  Importance  LI kely  Pessimistic  Optimistic  Expected  Deviation 


Attri tlon 

.014 

.035 

.01 

.017 

.004 

and 

10 

OJT  Time  Dollars 

1.0  wk 

1.5  wk 

.5  wk 

1 wk 

.167 

Trng.  Prog.  Dev. 

2 

Increase 

Flexlb. 

- 

- 

- 

- 

Standby  Equip. 

1 

Reduced 

30% 

- 

- 

- 

- 

User  Acceptance 
(w/o  projects) 

.50 

.40 

.60 

.50 

.033 

User  Acceptance 
(with  projects) 

.90 

.85 

.95 

.90 

.017 

Implementation 
Success  (w/o  projects) 

.75 

.65 

.85 

.75 

.033 

Implementation 
Success  (with  projects) 

.85 

.70 

.80 

.85 

.017 

Project  Success 

.95 

.90 

1.00 

.95 

.017 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 

PREPARE  COST  MODEL  CASES  AND  INPUT  DATA  (06.02) 

Develop  Outcome  Scenarios 

A - Attrition  will  be  reduced  from  .07  to  .01. 

I ns tr. /Stud.  Position  Ratio  will  go  from  .25  to  .05. 

Equipment  Life  will  go  from  5 years  to  10  years. 

Equipment  Unit  Cost  will  go  from  $4375  to  $5625. 

OJT  time  would  be  reduced  from  4 weeks  to  1 week. 

All  other  cost  data  from  the  formerly  planned  program  to  the 
proposed  program  will  remain  the  same. 

B - Attrition  could  fail  to  reach  .01  and  will  stay  around  .04. 

Two  years  will  be  required  to  determine  acceptance. 

Recovery  will  be  accomplished  by  reinstituting  the  baseline  or 
existing  plan  on  which  the  improvements  in  Outcome  A were  based. 

Trainers  will  have  no  salvage  value  at  end  of  two-year  period. 

No  OJT  time  will  be  saved. 

C - Approximately  $700K  in  development  costs  will  be  incurred. 

Approximately  $300K  will  be  required  to  expedite  equipment 
procurement  to  revert  to  existing  program  plan. 

No  OJT  time  will  be  saved. 

D - Neutral  Outcome. 

NOTE:  These  outcome  scenarios  are  highly  simpli- 

fied and  in  actual  practice  would  require 
considerably  more  analysis. 

Prepare  Cost  Model  Input  Data  Sheets 

• A run  of  the  existing  program  plan  is  made  by  preparing  input 
data  as  shown  in  Figure  V-4  and  V-5  for  the  Training  Cost  Model. 

• A run  of  the  proposed  program  is  made  by  preparing  input  data 
sheets  as  shown  in  Figure  V-6  and  V-7  for  the  Training  Cost 
Model . 
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COURSE  COST  ANALYSIS 


INPUT 


COURSE  NUMBER  - CTRL 

DISCOUNT  RATE  - 10. 

0 PERCENT 

PLANNING  PERIOO  - 8 

INFLATION  RATE  - 6. 

0 PERCENT 

INPUT  TECHNICAL  FACTORS: 

PLNG  TR  i 2 

3 

4 

5 

6 7 8 9 

10 

GRADS-  10000  12000 

14000  14000 

13000 

9000  8000  5000 

0 

0 

PLNG  YR  11  12 

13 

14 

15 

16  17  18  19 

20 

GRADS-  0 0 

0 0 

0 

0 0 0 

0 

0 

ATTRITION  RATE  (PCT) 

— 

0.07 

TIME 

STUD  POS  DOWN  (PCT)  - 

.01 

TRAINING  LENGTH  (WKS) 

- 

2.0 

PCT 

EXTRA  STUD  POSITIONS  - 

.0 

RECYCLE  RATE  (PCT) 

- 

.10 

EQU1 P/STUO  POS  RATIO 

0.25 

AVE.  RECYCLE  TIME  (WKS>- 

1.0 

LIFE 

OF  EQUIPMENT 

5 

WEEKS  SCHOOL  OPERATES 

- 

50.0 

COURSE  DEV  HRS/COURSE  HR  - 

25.0 

INSTR/STUO  POS  RATIO 

- 

0.25 

PCT 

COURSE  REQUIRING  DEV.- 

1.00 

ADMIN/STUO  POS  RATIO 

- 

.010 

PCT 

INSTR  MAT.  MAINTAINED- 

.20 

SO  FEET /STUDENT  POS. 

• 

9.00 

PCT 

INSTR  MAT.  RMNG  VALUE- 

.40 

SQ  FEET/ INSTR . POS. 

- 

5.00 

SUPPLIES/STUDENT  POSITION- 

1.0 

SQ  FEET/ADHIN  POS. 

- 

3.00 

SUPPLIES/S  TUOENT 

1.0 

PURCHASE  POLICY 

- 

2 

MISC/STUDENT  POSITION 

1.0 

DEPRECIATION  POLICY 

- 

2 

MISC/STUDENT 

1.0 

FIGURE  V-4.  TRAINING  COST  MODEL  TECHNICAL  INPUT  DATA  EXISTING  PROGRAM 
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*n-h 


COURSE  COST 

ANALYSIS 

INPUT: 

COURSE  NUMBER  - CTRL 

DISCOUNT 

RATE  - 

10.0 

PERCENT 

PLANNING  PERIOO  - 8 

INFLATION 

1 RATE  - 

6.0 

PERCENT 

INPUT  COST  FACTORS: 

PLNG  YR  l 2 

3 4 

5 

6 7 

8 

9 

10 

OCMN-  200.  200. 

200.  200. 

200. 

200.  200. 

200. 

0. 

0. 

PLNG  YR  11  12 

13  14 

15 

16  17 

18 

19 

20 

OGMN-  0.  0. 

• 

o 

• 

o 

0. 

o 

• 

o 

t 

0. 

0. 

0. 

STUOENT  SALARY  (ANNUAL! 

1-  11141.00 

COST/HR  OF  INSTR 

MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

228.00 

SUPPLIES  COST 

- 

10.00 

STUD  TRAVEL  IN  COURSE 

0.0 

MISCELLANEOUS  COST 

- 

0.0 

INSTR  SALARY  (ANNUAL) 

- 16240.00 

PCT 

MAX  EQUIP  PURCHASED 

— 

0.90 

ADMIN  SALARY  (ANNUAL) 

- 13500.00 

FACILITY  COST/SQ 

FOOT 

- 

25.00 

EQUIPMENT  UNIT  COST 

- 4375.00 

FIGURE  V-6.  TRAINING  COST  MODEL  COST  INPUT  DATA  - EXISTING  PROGRAM 
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COURSE 

COST  ANALYSIS 

INPUT: 

COURSE  NUMBER  - EXPM 

DISCOUNT  RATE  - 10. 

0 PERCENT 

PLANNING  PERIOD  - 8 

INFLATION  RATE  - 6. 

0 PERCENT 

INPUT  TECHNICAL  FACTORS: 

PLNG  YR  l 2 

3 

4 

5 6 7 8 9 

10 

GRADS-  IOOOO  12000 

14000 

14000 

13000  9000  8000  5000 

0 

o 

PLNG  YR  11  12 

13 

14 

15  16  17  18  19 

20 

GRADS-  0 0 

0 

0 

0 0 0 0 

0 

0 

ATTRITION  RATE  IPCT) 

— 

0. 

014 

TIME  STUD  POS  OOMN  (PCT)  - 

.01 

TRAINING  LENGTH  ( WKS ) 

- 

2. 

0 

PCT  EXTRA  STUD  POSITIONS  - 

.05 

RECYCLE  RATE  ( PCT ) 

- 

• 

10 

EQUI P/STUO  POS  RATIO 

0.50 

AVE.  RECYCLE  TIME  IWKS1- 

1. 

0 

LIFE  OF  EQUIPMENT 

10 

MEEKS  SCHOOL  OPERATES 

50. 

0 

COURSE  DEV  HRS/COURSE  HR  - 

25.0 

INSTR/STUD  POS  RATIO 

- 

0 

.05 

PCT  COURSE  REQUIRING  DEV.- 

1.00 

ADMIN/STUD  POS  RATIO 

- 

• 

010 

PCT  INSTR  MAT.  MAINTAINED- 

.20 

SO  FEET/STUDENT  POS. 

- 

9. 

00 

PCT  INSTR  MAT.  RMNG  VALUE- 

.40 

SQ  FEET/INSTR.  POS. 

- 

5. 

00 

SUPPLIES/STUDENT  POSITION- 

1.0 

SQ  FEET/AOMIN  POS. 

- 

3. 

00 

SUPPLIES/STUOENT 

1.0 

PURCHASE  POLICY 

- 

2 

MI  SC/STUDENT  POSITION 

1.0 

DEPRECIATION  POLICY 

1 

MISC/STUDENT 

1.0 

FIGURE  V-6.  TRAINING  COST  MODEL  TECHNICAL  INPUT  DATA  - PROPOSED  PROGRAM 
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COURSE  COST 

ANALYSIS 

INPUT: 

COURSE  NUMBER  - EXPM 

DISCOUNT  RATE  - 

10.0 

PERCENT 

PLANNING  PERIOD  - 8 

INFLATION  RATE  - 

6.0 

PERCENT 

INPUT  COST  FACTORS: 

PLNG  YR  l 2 

3 4 

5 6 7 8 

9 

10 

OCMN-  200.  200. 

200.  200. 

200.  200.  200.  200. 

0. 

0. 

PLNG  YR  li  12 

13  14 

15  16  17  18 

19 

20 

O&MN—  0.  0. 

• 

o 

t 

o 

o 

• 

o 

• 

0 

1 

O 

• 

0. 

0. 

STUDENT  SALARY  (ANNUAL) 

- 11141.00 

COST/HR  OF  INSTR  MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

228.00 

SUPPLIES  COST 

- 

10.00 

STUD  TRAVEL  IN  COURSE 

0.0 

MISCELLANEOUS  COST 

- 

0.0 

INSTR  SALARY  (ANNUAL) 

- 16240.00 

PCT  MAX  EQUIP  PURCHASED 

— 

0.90 

ADMIN  SALARY  (ANNUAL) 
EQUIPMENT  UNIT  COST 

- 13500.00 

- 5625.00 

FACILITY  COST/SQ  FOOT 

25.00 

FIGURE  V-7.  TRAINING  COST  MODEL  COST  INPUT  DATA  • PROPOSED  PROGRAM 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 


• Input  to  the  Job  Cost  Model  will  be  as  follows: 

3 weeks  OJT  time  saved. 

Annual  Salary  Cost  is  $11,441. 

Annual  numbers  of  graduates  used  in  the  input  to  the  Train- 
ing Cost  Model  will  be  the  basis  for  job  incumbents. 

• To  derive  costs  for  Outcome  B,  input  data  are  prepared  for  a 
two-year  run  with  data  for  the  proposed  program,  and  a six- 
year  run  with  data  for  the  existing  program  plan.  Examples 
of  input  data  sheets  to  accomplish  this  are  shown  In  Figures 
V-8  through  V-ll. 

NOTE:  The  financial  analysis  of  Task  7 will  use 

costs  and  savings  separated  by  R&D,  Invest- 
ment, or  Recurring.  To  save  possible  re- 
classification later,  this  categorization 
of  costs  should  be  considered  at  this  time. 

Ensure  True  "Opportunity  Costs"  Have  Been  Used 

All  costs  used  are  true  "opportunity  costs." 

The  basis  for  savings  in  job  incumbent's  time  was  straight  salary, 
exclusive  of  retirement  benefits,  etc. 

Input  Factors 

Time  saved  per  incumbent:  3 weeks/year. 

Number  of  incumbent  equals  annual  graduates. 

Salary  cost:  $11 ,141 . 

RUN  APPROPRIATE  COST  MODELS  (06.03) 

Training  Cost  Model  Outputs 

The  Training  Cost  Model  cost  output  data  for  the  existing  and  proposed 
programs  are  shown  in  Figure  V-12  and  V-13.  The  Discounted  Constant 
Dollar  value  for  each  program  is  shown  below,  and  the  Incremental  value 
calculated: 

OUTCOME  A 

Existing  $ 57,361,408  Cost 

Proposed  47,778,656  Cost 

Outcome  A Incr.  Training  $ 9,582,752  Savings 
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COURSE 

COST  ANALYSIS 

INPUT: 

COURSE  NUMBER  - RCVA 

DISCOUNT  RATE  - 

10. 

0 PERCENT 

PLANNING  PERIOD  - 2 

INFLATION  RATE  - 

6. 

0 PERCENT 

INPUT  TECHNICAL  FACTORS: 

PLNG  YR  1 2 

3 

A 

5 

6 7 8 

9 

10 

GRADS-  10000  12000 

0 

0 0 

0 0 0 

0 

0 

PLNG  YR  11  12 

13 

1 A 

15 

16  17  18 

19 

20 

GRADS-  0 0 

0 

0 0 

0 0 0 

0 

0 

ATTRITION  RATE  IPCT) 

— 

0.04 

TIME 

STUD  POS  DOWN  (PCT) 

— 

.01 

TRAINING  LENGTH  (WKS) 

- 

2.0 

PCT 

EXTRA  STUD  POSITIONS 

- 

.05 

RECYCLE  RATE  ( PCT ) 

- 

.10 

EQUIP/STUD  POS  RATIO 

- 

0.50 

AVE.  RECYCLE  TIME  INKS) 

• 

1.0 

LIFE 

OF  EQUIPMENT 

- 

10 

WEEKS  SCHOOL  OPERATES 

- 

50.0 

COURSE  DEV  HRS/COURSE  HR 

- 

25.0 

INSTR/STUD  POS  RATIO 

- 

0.05 

PCT 

COURSE  REQUIRING  DEV. 

- 

1.00 

ADMIN/STUD  POS  RATIO 

- 

.010 

PCT 

INSTR  MAT.  MAINTAINED- 

.20 

SQ  FEET/STUDENT  POS. 

- 

9.00 

PCT 

INSTR  MAT.  RMNG  VALUE 

• 40 

SQ  FEET/INSTR.  POS. 

- 

5.00 

SUPPLIES/STUDENT  POSITION- 

1.0 

SQ  FEET/ADHIN  POS. 

- 

3.00 

SUPPLI ES/STUOENT 

- 

1.0 

PURCHASE  POLICY 

- 

2 

MISC/STUDENT  POSITION 

- 

1.0 

DEPRECIATION  POLICY 

3 

MISC/STUDENT 

1.0 

> 

m 

tr> 

a 

-u 


ro 

i 

to 


FIGURE  V-f.  TRAINING  COST  MODEL  TECHNICAL  INPUT  DATA  - TWO  YEARS  WITH  PROPOSED  PROGRAM 


COURSE  COST 

ANALYSIS 

INPUT: 

COURSE  NUMBER 

- RCVA 

DISCOUNT 

RATE  - 

10.0 

PERCENT 

PLANNING  PERIOD 

- 2 

INFLATION 

RATE  - 

6.0 

PERCENT 

INPUT  COST  FACTORS: 

PLNG  YR  1 

2 

3 A 

5 6 7 

8 

9 

10 

OCMN-  200. 

200. 

• 

o 

• 

o 

O 

t 

o 

• 

o 

• 

0. 

0. 

0. 

PLNG  YR  11 

12 

13  14 

15  16  17 

18 

19 

20 

OCMN-  0. 

0. 

0.  0. 

o 

• 

o 

• 

o 

• 

0. 

0. 

0. 

STUDENT  SALARY 

(ANNUAL ) 

- 11141.00 

COST/HR  OF  INSTR 

MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

228.00 

SUPPLIES  COST 

- 

10.00 

STUD  TRAVEL  IN 

COURSE 

0.0 

MISCELLANEOUS  COST 

- 

0.0 

INSTR  SALARY  (ANNUAL) 

- 16240.00 

PCT  MAX  EQUIP  PURCHASED 

- 

0.90 

ADMIN  SALARY  (ANNUAL) 

- 13500.00 

FACILITY  COST/SQ 

FOOT 

- 

25.00 

EQUIPMENT  UNIT 

COST 

- 5625.00 

FIGURE  V-9.  TRAINING  COST  MODEL  COST  INPUT  DATA  - TWO  YEARS  WITH  PROPOSED  PROGRAM 
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COURSE 

COST  ANALYSIS 

INPUT: 

COURSE  NUMBER  - RCVB 

DISCOUNT  RATE  - 10. 

0 PERCENT 

PLANNING  PERIOD  - 8 

INFLATION  RATE  - 6. 

0 PERCENT 

INPUT  TECHNICAL  FACTORS: 

PLNG  YR  1 2 

3 

4 

5 

6 7 8 9 

10 

GRADS-  0 0 

14000 

14000 

13000 

9000  8000  5000 

0 

0 

PLNG  YR  11  12 

13 

14 

15 

16  17  18  19 

20 

GRADS-  0 0 

0 

0 

0 

0 0 0 

0 

0 

ATTRITION  RATE  (PCT) 

— 

• 

07 

TIME 

STUD  POS  DOWN  (PCT)  - 

.01 

TRAINING  LENGTH  (WKS) 

- 

2. 

0 

PCT 

EXTRA  STUD  POSITIONS  - 

.0 

RECYCLE  RATE  (PCT) 

- 

• 

10 

EQUIP/STUD  POS  RATIO 

0.25 

AVE.  RECYCLE  TIME  (WKS)- 

1. 

0 

LIFE 

OF  EQUIPMENT 

5 

WEEKS  SCHOOL  OPERATES 

- 

50. 

0 

COURSE  DEV  HRS/COURSE  HR  - 

25.0 

INSTR/STUD  POS  RATIO 

- 

• 

250 

PCT 

COURSE  REQUIRING  DEV.- 

1.00 

ADMIN/STUD  POS  RATIO 

- 

• 

010 

PCT 

INSTR  HAT.  HAINTAINED- 

.20 

SQ  FEET/STUDENT  POS. 

- 

9. 

00 

PCT 

INSTR  MAT.  RMNG  VALUE— 

.40 

SQ  FEET/INSTR.  POS. 

- 

5. 

00 

SUPPLIES/STUDENT  POSITION- 

1.0 

SQ  FEET/ADMIN  POS. 

- 

3. 

00 

SUPPLIES/STUDENT 

1.0 

PURCHASE  POLICY 

- 

2 

MISC/STUDENT  POSITION 

1.0 

DEPRECIATION  POLICY 

2 

MISC/STUDENT 

1.0 

FIGURE  V-10.  TRAINING  COST  MODEL  TECHNICAL  INPUT  DATA  - SIX  YEARS  WITH  EXISTING  PROGRAM 
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COURSE  COST 

ANALYSIS 

INPUT; 

COURSE  NUMBER 

- RCVB 

DISCOUNT 

RATE  - 

10.0 

PERCENT 

PLANNING  PER IOO 

- 8 

INFLATION 

RATE  - 

6.0 

PERCENT 

INPUT  COST  FACTORS: 

PLNG  YR  I 

2 

3 A 

5 6 7 

8 

9 

10 

OCMN-  200. 

200. 

200.  200. 

200.  200.  200. 

200. 

0. 

0. 

PLNG  YR  11 

12 

13  1 A 

15  16  17 

18 

19 

20 

OCMN-  0. 

0. 

• 

o 

• 

o 

• 

o 

t 

o 

• 

o 

0. 

0. 

0. 

STUDENT  SALARY 

(ANNUAL 

)-  111A1.00 

COST/HR  OF  INSTR 

MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

228.00 

SUPPLIES  COST 

- 

10.00 

STUD  TRAVEL  IN 

COURSE 

0.0 

MISCELLANEOUS  COST 

- 

0.0 

INSTR  SALARY  (ANNUAL) 

- 162A0.00 

PCT  MAX  EQUIP  PURCHASED 

- 

.90 

AOMIN  SALARY  (ANNUAL) 

- 13500.00 

FACILITY  COST/SQ 

FOOT 

- 

25.00 

EQUIPMENT  UNIT 

COST 

- A375.00 

FIGURE  V-11.  TRAINING  COST  MODEL  COST  INPUT  DATA  - SIX  YEARS  WITH  EXISTING  PROGRAM 
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COURSE  COST  ANALYSIS 


INPUT: 

COURSE  NUMBER  - CTRL 

PLANNING  PERIOD  - 8 

OUTPUT  COST  RESULTS: 

NON-DISC 

STUDENT  SALARIES 

CONSTANT  $ 
39751344. 

STUDENT  TRAVEL 

20838672. 

INSTRUCTOR  SALARY 

15217819. 

ADMINISTRATIVE  SALARY 

506011. 

EQUIPMENT 

566900. 

FACILITIES 

963295. 

INSTRUCTIONAL  MATERIALS 

14400. 

SUPPLIES 

951460. 

MISCELLANEOUS 

0. 

TOTAL  OPERATIONAL 

78809840. 

COST  PER  GRADUATE 

927.17 

COST  PER  STUDENT  INPUT 

862.27 

COST  PER  STUDENT  POSITION 

21049.64 

DISCOUNT  RATE  - 10. 0 PERCENT 
INFLATION  RATE  - 6.0  PERCENT 


DISCOUNTED 

NON-OISC 

DISCOUNTED 

CONSTANT  $ 

CURRENT  $ 

CURRENT  S 

28814160. 

49306016. 

34934912. 

151051 19. 

25847472. 

18313776. 

11030770. 

18875568. 

13373961. 

366786. 

627636. 

444700. 

645611. 

437360. 

611993. 

698253. 

1194832. 

846578. 

11093. 

17018. 

12989. 

689675. 

1180153. 

836177. 

0. 

0. 

0. 

57361408. 

97486016. 

69375040. 

674.84 

1146.89 

816.18 

627.60 

1066.61 

759.05 

15320.89 

26037.93 

18529.66 

FIGURE  V-12.  TRAINING  COST  MODEL  COST  OUTPUT  DATA  - EXISTING  PROGRAM 
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DISCOUNT  RATE  - 10.0  PERCENT 
INFLATION  RATE  - 6.0  PERCENT 


INPUT: 

COURSE  NUMBER  - EXPM 
PLANNING  PERIOD  - 8 

OUTPUT  COST  RESULTS: 

STUDENT  SALARIES 
STUDENT  TRAVEL 
INSTRUCTOR  SALARY 
ADMINISTRATIVE  SALARY 
EQUIPMENT 
FACILITIES 

INSTRUCTIONAL  MATERIALS 

SUPPLIES 

MISCELLANEOUS 

TOTAL  OPERATIONAL 

COST  PER  GRADUATE 
COST  PER  STUDENT  INPUT 
COST  PER  STUDENT  POSITION 


COURSE  COST  ANALYSIS 


NON-DISC 

DISCOUNTED 

CONSTANT  * 

CONSTANT  t 

38527984. 

27927344. 

19655152. 

14247222. 

3097385. 

2245167. 

514959. 

373273. 

1726300. 

1680617. 

884968. 

641477. 

14400. 

11093. 

900214. 

652528. 

0. 

0. 

65321296. 

47778656. 

768.49 

562.10 

757.73 

554.23 

17140.20 

12537.04 

NON-DISC 

DISCOUNTED 

CURRENT  $ 

CURRENT  $ 

47788608. 

33859776. 

24379456. 

17273648. 

3841873. 

2722092. 

638735. 

452565. 

1722088. 

1714579. 

1097678. 

777741. 

17018. 

12989. 

1116589. 

791140. 

0. 

0. 

80601984. 

57604480. 

948.26 

677.70 

934.98 

668.21 

21149.82 

15115.32 

FIGURE  V-13.  TRAINING  COST  MODEL  COST  OUTPUT  DATA  - PROPOSED  PROGRAM 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 


The  cost  output  data  for  Outcome  B are  shown  In  Figure  V-14  and  V-15 
and  are  totaled  below. 

OUTCOME  B 


Proposed  (2  years) 
Existing  (6  years) 

$ 

16,614,874 

38,992,672 

Cost 

Cost 

Outcome  B Total  Training 

$ 

55,607,546 

Cost 

The  incremental  value  calculated  between 
proposed  program,  considering  Outcome  B, 

the 

Is: 

existing  program  and 

Existing 
Outcome  B Total 

$ 

57,361,408 

55,607,546 

Cost 

Cost 

Outcome  B Incr.  Training 

$ 

1,753,862 

Savings 

OUTCOME  C 

Dev.  Costs 
Expedite  Costs 

$ 

700.000 

300.000 

Cost 

Cost 

Outcome  C Incr.  Training 

$ 

1,000,000 

Cost 

Job  Cost  Model  Outputs 

Only  the  results  are  shown,  since  no  Job  Cost  Model  was  available. 
OUTCOME  A 

4904  man-years  and  $37,086,976  savings. 

All  other  outcomes  are  zero. 

DETERMINE  BENEFIT-LIABILITY  VALUES  (06.04) 

Use  Variable  Data  Assembled  in  06.01 

Variable  Importance 

Standby  parts  and  equipment  reduced  in  Fleet  1 

Flexibility  In  program  development  2 

Rank  Outcome  for  Each  Variable 


The  outcomes  are  ranked  as  shown  in  the  first  three  columns  of  Figure 
V-16  and  V-17.  In  both  cases.  Outcome  D (To  Reject  the  Innovation) 
is  considered  neutral. 
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INPUT: 

COURSE  NUMBER  - RCVA 

PLANNING  PERIOD  - 2 

COURSE  COST 

ANALYSIS 

DISCOUNT  RATE  - 
INFLATION  RATE  - 

10. 0 PERCENT 
6.0  PERCENT 

OUTPUT  COST  RESULTS: 

NON-DISC 
CONSTANT  $ 

DISCOUNTED 
CONSTANT  $ 

NON-DISC 
CURRENT  S 

DISCOUNTED 
CURRENT  $ 

STUDENT  SALARIES 

10114376. 

9175896. 

10749610. 

9738751. 

STUDENT  TRAVEL 

5224998. 

4740187. 

5553154. 

5030953. 

INSTRUCTOR  SALARY 

813126. 

737678. 

864194. 

782928. 

ADMINISTRATIVE  SALARY 

135187. 

122644. 

143678. 

130167. 

EQUIPMENT 

1482150. 

1410510. 

1528354. 

1454226. 

FACILITIES 

232322. 

210765. 

246912. 

223694. 

INSTRUCTIONAL  MATERIALS 

0. 

208. 

-148. 

86. 

SUPPLIES 

239180. 

216988. 

254202. 

230298. 

MISCELLANEOUS 

0. 

0. 

0. 

0. 

TOTAL  OPERATIONAL 

18241328. 

16614874. 

19339904. 

17591088. 

COST  PER  GRADUATE 

829.15 

755.22 

879.09 

799.59 

COST  PER  STUDENT  INPUT 

795.99 

725.01 

843.92 

767.61 

COST  PER  STUDENT  POSITION 

18223.10 

16598.27 

19320.58 

17573.51 

FIGURE  V-14.  TRAINING  COST  MODEL  COST  OUTPUT  DATA  - TWO  YEARS  WITH  PROPOSED  PROGRAM 
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INPUTS 

COURSE  NUMBER  - RCVB 

COURSE  COST 

ANALYSIS 

DISCOUNT  RATE  - 

10.0  PERCENT 

PLANNING  PERIOD  - 8 

INFLATION  RATE  - 

6.0  PERCENT 

OUTPUT  COST  RESULTS: 

NON-DISC 

DISCOUNTED 

NON-DISC 

DISCOUNTED 

CONSTANT  t 

CONSTANT  % 

CURRENT  $ 

CURRENT  $ 

STUDENT  SALARIES 

29462768. 

19480208. 

38371264. 

25028416. 

STUDENT  TRAVEL 

15445156. 

10212023. 

20115184. 

13120553. 

INSTRUCTOR  SALARY 

11279089. 

7457503. 

14689476. 

9581508. 

ADMINISTRATIVE  SALARY 

375043. 

247971. 

488443. 

318597. 

EQUIPMENT 

566900. 

645611. 

437360. 

611993. 

FACILITIES 

713972. 

472064. 

929851. 

606514. 

INSTRUCTIONAL  MATERIALS 

14400. 

11093. 

17018. 

12989. 

SUPPLIES 

705200. 

466264. 

918427. 

599063. 

MISCELLANEOUS 

0. 

0. 

0. 

0. 

TOTAL  OPERATIONAL 

58562496. 

38992672. 

75966960. 

49879584. 

COST  PER  GRADUATE 

929.56 

618.93 

1205.82 

791.74  | 

COST  PER  STUDENT  INPUT 

864.49 

575.61 

1121.42 

736.32 

COST  PER  STUDENT  POSITION 

21103.60 

14051.41 

27375.48 

17974.62 

FIGURE  V-15.  TRAINING  COST  MODEL  COST  OUTPUT  DATA  - SIX  YEARS  WITH  EXISTING  PROGRAM 
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FIGURE  V-16.  BENEFIT  RANKING  AND  EVALUATION: 
STANDBY  PARTS  AND  EQUIPMENT  REDUCED  IN  FLEET 
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FIGURE  V-17.  BENEFIT  RANKING  AND  EVALUATION: 
FLEXIBILITY  IN  TRAINING  PROGRAM  DEVELOPMENT 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 

Scale  the  Ranked  Outcomes 

The  first  and  last  ranked  outcomes  become  100  and  0,  respectively, 
as  shown  in  Figures  V-16  and  V-17. 

Benefit:  Standby  parts  and  equipment  reduced  in  Fleet 

A = 100 

B = 0 

D is  neutral 

A is  four  times  more  favorable  than  B is  unfavorable,  therefore, 
the  scale  from  A to  D is  four  times  the  distance  D to  B. 

D = 20 

C is  only  half  as  unfavorable  as  B 
C = 10 

(See  Figure  V-16.) 

Benefit:  Flexibility  in  training  program  development 

This  follows  the  same  logic  as  for  the  previous  benefit  analyzed. 
(See  Figure  V-17  for  results.) 

REFINE  SUCCESS  PROBABILITIES  (06.05  - Use  04.02  and  04.04) 

The  reanalysis  of  the  success  probabilities  shows  no  change  from  those 
estimated  in  Task  4 and  assembled  as  input  to  Task  6 in  06.01. 

User  Acceptance  (w/o  projects)  50%  (40%  to  60%) 

User  Acceptance  (with  projects)  90%  (85%  ot  95%) 

Implementation  Success  (w/o  projects)  75%  (65%  to  85%) 

Implementation  Success  (with  projects)  85%  (70%  to  80%) 

Project  Success  95%  (90%  to  100%) 

If  projects  fail,  use  user  acceptance  and  implementation  success  w/o 
projects  factors. 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 

DEVELOP  DECISION  TREE  ASSESSMENT  (06.06) 


Data  Requirements 

Cost  model  incremental  outputs  from  06.03. 

Benefits  ranking  and  valuation  data  from  06.04. 

Refined  success  probabilities  from  06.05. 

Develop  Equivalent  Dollars  for  Nonquantified  Benefits 
Standby  equipment  variable. 

The  importance  ratio  of  1:10  causes  this  benefit  scale  of  0-100 
to  be  adjusted  as  shown  In  Figure  V-18. 

Flexibility  in  Training  Program  Development 

(See  results  on  Figure  V-18.) 

NOTE:  The  sum  of  the  ATTRITION  and  OJT  TIME 

benefits  measured  in  actual  dollars, 
was  used  as  the  basis  for  scaling  the 
two  nonquantified  variables. 

Display  Range  of  Values  for  Each  Outcome 

Figure  V-19  shows  the  relationship  between  the  valuations  for  each 
benefit,  adjusted  for  importance.  The  total  value  (in  equivalent 
dollars)  for  each  outcome  Is  summarized  at  the  bottom  of  the  chart. 
These  values  are  entered  directly  onto  the  decision  tree  for  Outcomes 
A through  D. 

Complete  Decision  Tree  Assessment  Diagram 

The  values  from  Figure  V-19  for  each  outcome  are  entered  onto  the  de- 
cision tree,  as  are  the  values  for  the  readjusted  success  probabilities 
from  06.05.  Figure  V-20  shows  a completed  decision  tree  diagram. 

Foldback  the  Decision  Tree 


The  folded  back  decision  tree  of  Figure  V-20  confirms  more  strongly, 
the  indicated  decision  tree  path  established  in  Task  4,  using  the 
preliminary  range-of-effect  data.  The  calculated  decision  variable 
is  32.5,  favoring  acceptance  and  undertaking  of  the  risk  reduction 
projects. 
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RELEVANT 

RELATIVE 

. VARIABLE 

IMPORTANCE 

A 

B 

C 

D 

COMMENTS 

ATTRITION 

9.6 

1.8 

-1.0 

0.0 

TOTAL  SINGLE 

VALUE  IS 

47.7  (46.7  + 

1.0)  AND 

OJT  TIME 

a 

37.1 

0.0 

0.0 

0.0 

WILL  CARRY  A 
OF  10. 

WEIGHT 

TOTAL 

10 

46.7 

1.8 

0.0 

STANDBY  EQUIP 

1 

100 

0 

10 

20 

SCALE  VALUES 

FROM  FIG 

URE  V-16.  D IS  THE 
NEUTRAL  OUTCOME  AND 
THE  TOTAL  SCALE  WILL 
BE  1/10  OF  47.7  OR 
4.8. 


ADJUSTED  3.8  -1.0  -.5  0.0  SCALE  VALUES  FROM  FIG- 

URE V-16.  ADJUST  CON- 
SIDERING THE  RELATIVE 
IMPORTANCE  OF  THIS 
BENEFIT  COMPARED  TO 
THE  IMPORTANCE  OF  TOT- 
AL REAL  DOLLARS  SAVED. 


FLEXIBILITY  IN  2 100 

TRAINING  PROGRAM 
DEVELOPMENT 


4 0 10  SCALE  VALUES  FROM  FIG- 

URE V-17.  D IS  THE 
NEUTRAL  OUTCOME  AND 
THE  TOTAL  SCALE  WILL 
BE  2/10  OF  47.7  OR 
9.5. 


ADJUSTED  8.5  -.4  -1.0  0.0  SCALE  VALUES  FROM  FIG- 

URE V-17.  ADJUSTED 
CONSIDERING  RELATIVE 
IMPORTANCE  OF  THIS 
BENEFIT  COMPARED  TO 
THE  IMPORTANCE  OF  TOT- 
AL REAL  DOLLARS  SAVED. 


FIGURE  V-18.  BENEFIT  OUTCOME  SUMMARY 
(3-D  PROCEDURAL  TRAINER) 
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OUTCOME  ANALYSIS  - BENEFIT  UTILITY  ASSIGNMENT 


ASSESSMENT 

DIMENSIONS 


oo 

Z 

o 


oo 

CL 

3 

O 

o 


< 

> 

B 

0 

Ui 

1 

00 

UJ 

2E 

O 

o 

I- 
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O 

u. 
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< 
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OUTCOME 

SUMMARY 


+ 10 

+ 9 
+ 8 
+ 7 
+ 6 
+ 5 
+ 4 
+ 3 
+ 2 
+ 1 
0 
1 

• 2 

• 3 

- 4 

- 5 

- 6 

• 7 

• 8 
- 9 
-10 


46.7 


1.8 


1.0 


0.0 


ACTUAL 
DOLLARS  FROM 
ATTRITION 
&OJTTIME 
IMPORTANCE  10 

STANDBY 

EQUIPMENT 

IMPORTANCE  1 

FLEXIBILITY 
TRNG  PROG 
DEVELOPM'T 

IMPORTANCE  2 

46.7  ' 

A 

i i 

- A 

! i 

1 1 

1 1 

| . 

l 1 

1 

— v 

A 

_l 

~T~ 

1 

1 

“a  | 

1 

0 

D | 

D 

c 1 

.U  I 

_ B 1 

P B 

_c 

1 

1 

: i 

1 

1 

i ! 

i ; 

• i 

1 i 

i i 

i i 

3.8 


1.0 


0.0 


8 J5 


.4 


1.0 


0.0 


TOTAL 


59.0 


.4 


2.5 


0.0 


FIGURE  V-19.  BENEFIT  OUTCOME  SUMMARY 
(3-D  PROCEDURAL  TRAINER  ANALYZED  OVER  FULL  RANGE-OF-EFFECT) 
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IMPORTANT  OUTCOMES 


A lAVIHCi 'BENEFITS 
ACCRUE 

• COSTS  • POSSIBLE 

LIABILITIES  RESULT 

C COSTS  ACCRUE 

0 NEUTRAL  OUTCOME 

A SAVINGS/BENE  FITS 
ACCRUE 

8 COSTS  POSSIBLE 

LIABILITIES  RESULT 

C COSTS  ACCRUE 

O NEUTRAL  OUTCOME 

A SAVINGS*  ENE  FITS 
ACCRUE 

B COSTS  POSSIBLE 
LIABILITIES  RESULT 

C COSTS  ACCRUE 

O NEUTRAL  OUTCOME 


FIGURE  V-20.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
(3-D  PROCEDURAL  TRAINER  ANALYZED  OVER  TOTAL 
RANGE-OF-EFFECT  INCLUDING  NONQUANTIFIED  BENEFITS) 
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EXHIBIT:  COST-BENEFITS  ANALYSIS  (Continued)... 


PERFORM  SENSITIVITY  ANALYSIS  (06.07) 

Review  Sensitivity  Profile  for  Each  Model 

The  characteristics  of  each  cost  model  are  reviewed  to  determine  if 
any  of  the  variables  used  in  the  preceding  analysis  might  be  sensi- 
tive enough  to  cause  a change  in  the  decision. 

Variable  Confidence  Range 

OJT  Time  1.5  to  .5  weeks 

Attrition  3.5%  to  1% 

User  Acceptance  (w/o  projects)  40%  to  60% 

(An  output  report  as  shown  in  the  06.07  procedures 
was  not  generated,  since  none  of  the  variables  caused 
a change  in  the  decision  across  the  confidence  range 
— scenario  values  were  selected  to  avoid  sensitivity 
calculations  because  of  excessive  manual  computational 
requirements. ) 

TASK  7 - PERFORM  FINANCIAL  ANALYSIS.  The  cost-benefits  analysis  has  In- 
dicated a decision  In  favor  of  accepting  the  Innovation,  including  the 
performance  of  the  risk  reduction  projects  identified  in  Task  4.  The 
techniques  of  financial  analysis  are  fairly  straightforward,  therefore, 
only  the  results  of  basic  comparison  between  the  existing  system,  the 
proposed  approach,  and  one  alternative  approach  identified  in  Task  2 are 
presented  here. 

EXHIBIT:  FINANCIAL  ANALYSIS 

ASSEMBLE  DATA  (07.01) 

Cost  Model  Output  Data  From  06,03 

Risk  reduction  project  costs. 

Training  savings. 

OJT  time  savings. 

Alternative  Approach  to  Meeting  Objective  From  Task  2 
DEVELOP  COST/SAVINGS  ANALYSIS  (07.02) 

Make  Required  Cost  Model  Runs 

An  additional  model  run  was  required  to  determine  the  cost  of  the  alter- 
native Identified  In  Task  2.  Figures  V-21  through  V-23  show  the  inputs 
and  outputs  for  this  run. 
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INPUT: 

COURSE  NUMBER  - ALT  I 
PLANNING  PER IOO  - 8 


COURSE  COST  ANALYSIS 


DISCOUNT  RATE 
INFLATION  RATE 


10.0  PERCENT 
6.0  PERCENT 


INPUT  TECHNICAL  FACTORS: 


PLNG  YR  1 

GRADS-  10000 

2 

12000 

3 

14000 

4 

14000 

5 

13000 

6 

9000 

7 

8000 

8 

5000 

9 

10 

0 

PLNG  YR  11 
GRADS-  0 

12 

0 

13 

0 

14 

0 

15 

0 

16 

0 

17 

0 

18 

0 

19 

20 

0 

ATTRITION  RATE 

(PCT) 

— 

.05 

TIME 

STUD 

POS  DOWN 

(PCT) 

— 

.01 

TRAINING  LENGTH  ( WKS)  - 2.0 

RECYCLE  RATE  (PCT)  - .10 

AVE.  RECYCLE  TIME  (WKS)-  1.0 

WEEKS  SCHOOL  OPERATES  - 50.0 

INSTR/STUD  POS  RATIO  - .200 

ADMIN/STUO  POS  RATIO  - .010 

SO  FEET/STUOENT  POS.  - 9.00 

SQ  FEET/INSTR.  POS.  - 5.00 

SQ  FEET/ADMIN  POS.  - 3.00 

PURCHASE  POLICY  - 2 

DEPRECIATION  POLICY  - 2 


PCT  EXTRA  STUD  POSITIONS  - .0 

EQUIP/STUD  POS  RATIO  - 0.20 

LIFE  OF  EQUIPMENT  - 5 

COURSE  DEV  HRS/COURSE  HR  - 25.0 

PCT  COURSE  REQUIRING  DEV.-  1.00 

PCT  INSTR  MAT.  MAINTAINED-  .20 

PCT  INSTR  MAT.  RMNG  VALUE-  .40 

SUPPLIES/STUDENT  POSITION-  1.0 

SUPPLIES/STUDENT  - 1.0 

MISC/STUDENT  POSITION  - 1.0 

MISC/STUDENT  - 1.0 


FIGURE  V-21.  TRAINING  COST  MODEL  TECHNICAL  INPUT  DATA  - ALTERNATIVE  IDENTIFIED  IISTTASK  2 
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COURSE  COST  ANALYSIS 


INPUTS 

COURSE  NUMBER 


- ALT  l 


DISCOUNT  RATE  - 10.0  PERCENT 


PLANNING 

PERIOD 

- 8 

INFLATION 

RATE  ~ 

6.0 

PERCENT 

INPUT  COST 

FACTORS: 

PLNG  YR 

i 

2 

3 

4 

5 

6 7 

e 

9 

10 

OGMN- 

200. 

200. 

200. 

200. 

200. 

200.  200. 

200. 

0. 

0. 

PLNG  YR 

11 

12 

13 

14 

15 

16  17 

16 

19 

20 

O&MN- 

0. 

0. 

0. 

0. 

0. 

0.  0. 

0. 

0. 

0. 

STUDENT  SALARY 

(ANNUAL) 

- 11141.00 

COST/HR  OF  INSTR 

MAT  DEV 

— 

6.00 

STUD  TRAVEL  TO/FROM 

- 

228.00 

SUPPLIES  COST 

— 

10.00 

STUD  TRAVEL  IN 

COURSE 

— 

0.0 

MISCELLANEOUS  COST 

— 

0.0 

INSTR  SALARY  (ANNUAL) 

- 16240.00 

PCT 

MAX  EQUIP  PURCHASED 

- 

.90 

ADMIN  SALARY  (ANNUAL) 

- 13500.00 

FACILITY  COST/SQ 

FOOT 

- 

25.00 

EQUIPMENT  UNIT  COST 


- 5000.00 


FIGURE  V-22.  TRAINING  COST  MODEL  COST  INPUT  DATA  - ALTERNATIVE  IDENTIFIED  IN  TASK  2 
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INPUT: 

COURSE  NUMBER  - ALT  1 

PLANNING  PERIOD  - 6 

COURSE  COST 

ANALYSIS 

DISCOUNT  RATE  - 
INFLATION  RATE  - 

10.0  PERCENT 
6.0  PERCENT 

OUTPUT  COST  RESULTS: 

NON-DISC 
CONSTANT  S 

DISCOUNTED 
CONSTANT  S 

NON-DISC 
CURRENT  S 

DISCOUNTED 
CURRENT  t 

STUDENT  SALARIES 

39297904. 

2 8485424. 

48743536. 

34536384. 

STUDENT  TRAVEL 

20399952. 

147871 15. 

25303312. 

17928224. 

INSTRUCTOR  SALARY 

12035370. 

8723944. 

14928194. 

10577111. 

ADMINISTRATIVE  SALARY 

500239. 

362602. 

620476. 

439627. 

EQUIPMENT 

489600. 

567772. 

365878. 

533536. 

FACILITIES 

929147. 

673500. 

1152476. 

816567. 

INSTRUCTIONAL  MATERIALS 

14400. 

11093. 

17018. 

12989. 

SUPPLIES 

931791. 

675417. 

1155756. 

818891. 

miscellaneous 

0. 

0. 

0. 

0. 

TOTAL  OPERATIONAL 

74598336. 

54286800. 

92286592. 

65663280. 

COST  PER  GRADUATE 

877.63 

638.67 

1085.72 

772.51 

COST  PER  STUDENT  INPUT 

833.75 

606. 74 

1031.44 

733.89 

COST  PER  STUDENT  POSITION 

20156.26 

14668. 14 

24935.58 

17742.04 

FIGURE  V-23.  TRAINING  COST  MODEL  COST  OUTPUT  DATA  - ALTERNATIVE  IDENTIFIED  IN  TASK  2 
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EXHIBIT:  FINANCIAL  ANALYSIS  (Continued)... 


Generate  Cost/Savings  Reports 

Each  alternative  is  assessed  for  incremental  effects  against  each 
other  alternative,  and  up  to  three  formats  of  Cost/Savings  Report 
are  generated  for  each  case. 

Cost/Savings  Reports  for  the  existing  vs.  the  proposed  system  have 
been  generated  in  two  formats  as  shown  in  Figure  V-24  and  V-25. 
Similar  reports  in  all  formats  would  normally  be  generated  for  the 
existing  vs.  the  alternative  system  and  for  the  proposed  vs.  the 
alternative  system. 

DEVELOP  ALTERNATIVES  COMPARISON  ANALYSIS  (07.03) 

Select  Alternatives  for  Further  Consideration 


Each  of  the  two  alternatives  (the  proposed  3-D  Procedural  Trainer  and 
the  alternative  from  Task  2)  has  been  selected  for  comparative  finan- 
cial analysis. 

Organize  Data 

Since  the  data  have  been  divided  previously  into  R&D,  Investment,  and 
Recurring,  processing  can  take  place  directly.  The  R&D  costs,  for 
economic  analysis  purposes,  are  treated  as  an  Investment,  as  are 
equipment  purchases. 

Generate  Alternatives  Analysis  Report 

Figure  V-26  summarizes  the  data  comparing  the  proposed  3-D  Procedural 
Trainer  with  the  existing  (or  presently  planned)  system,  the  alterna- 
tive approach  from  Task  2 with  the  existing  system,  and  the  3-D  Trainer 
with  the  alternative. 

Review  A1 ternati ves 


The  results  portrayed  in  the  Alternatives  Analysis  Report  may  be  diff- 
icult to  interpret  because  of  the  values  used  in  this  scenario.  They 
can  be  sunmarized  as  follows: 

• The  proposed  innovation  provides  about  a 75%  return  on  invest- 
ment which  is  extremely  high,  and  in  the  absence  of  excessive 
risks  (already  eliminated  with  the  risk  reduction  projects), 
should  be  accepted. 

• The  alternative  proposal,  however,  is  even  more  lucrative  since 
it  requires  a negative  investment  to  produce  savings.  This  is 
due  to  a lesser  equipment  cost  and  the  fact  that  no  R&D  funds 
were  identified  to  be  expended.  Thus,  it  would  be  favored  for 
acceptance  over  the  proposed  3-D  Procedural  Trainer. 
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COST/SAVINGS  SOURCE  REPORT 
(EXISTING  VS.  PROPOSED  SYSTEM) 


INPUT: 

DISCOUNT  RATE  - 10  PERCENT 
INFLATION  RATE  - 6 PERCENT 


DISPLAY  FORMAT  - COURSE/EQUIPMENT/ JOB 
PLANNING  PERIOD  - 08  YEARS 


OUTPUT: 

CST(-)/SAV(+)  SOURCE 
PROJECTS 
(LIST) 

COURSES 

(LIST) 

JOBS 

(LIST) 

NONDISCOUNTED  TOTAL  $ 
PRESENT  VALUE-CONST  $ 
INFLATED  TOTAL  $ 
PRESENT  VALUE rCRNT  $ 


TOTAL  YEAR  1 
- 11,123.2 
12,611.7 

54,635.5 

56.124.0 

35.727.0 

69.424.0 
49,615.8 


YEAR  2 YEAR  3 YEAR  4 YEAR  5 


(ANNUAL  INCREMENTAL  VALUES  NOT  CALCULATED) 


FIGURE  V-24.  COST/SAVINGS  ANALYSIS  OF  EXISTING  VS.  PROPOSED 
SYSTEM  BY  COURSE/EQUIPMENT/JOB  CATEGORY 
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OS -A 


COST/SAVINGS  SOURCE  REPORT 
(EXISTING  VS.  PROPOSED  SYSTEM) 


INPUT: 

DISCOUNT  RATE  - 10  PERCENT 
INFLATION  RATE  - 6 PERCENT 


DISPLAY  FORMAT  - COST  CATEGORIES 
PLANNING  PERIOD  - 08  YEARS 


OUTPUT: 

CST(-)/SAV(+)  SOURCE  TOTAL  YEAR  1 
R&D 


(LIST) 

-11,123.2 

INVESTMENT 

(LIST) 

- 2,036.2 

RECURRING 

(list) 

69,283.4 

NONDISCOUNTED  TOTAL 

$ 

56,124.0 

PRESENT  VALUE-CONST 

$ 

35,727.0 

INFLATED  TOTAL 

$ 

69,424.0 

PRESENT  VALUE -CRNT 

$ 

49,615.8 

YEAR  2 YEAR  3 YEAR  4 YEAR  5 


(ANNUAL  INCREMENTAL  VALUES  NOT  CALCULATED) 


FIGURE  V-25.  COST/SAVINGS  ANALYSIS  OF  EXISTING  VS.  PROPOSED 
SYSTEM  BY  COST  CATEGORY 
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ALTERNATIVES  ANALYSIS  REPORT 


SUMMARY  OF  RELEVANT  FACTORS  CONSIDERED 
IN  THE  ECONOMIC  ANALYSES: 

PROPOSED  VS. 
EXISTING 

ALTERNATIVE 
VS.  EXISTING 

PROPOSED  VS 
ALTERNATIVE 

COMPARATIVE  ECONOMIC  LIFE  OF 

8 V 

8 V 

8 V 

ALTERNATIVES  (YEARS) 

S 8 

S 

8 

S 8 

TOTAL  NONDISC  NET  INVESTMENT 

13,159.4  - 

77.3 

+ 

13,236.7  - 

TOTAL  NONDISC  RECURRING  CST/SAV 

69,283.4 

58,769.7 

10,513.7 

PRESENT  VALUE  OF  NEW  INVESTMENT 

11,977.1  - 

77.8 

+ 

12,054.9  - 

PLUS:  PV  EXISTING  ASSETS  USED 

0.0 

0.0 

0.0 

LESS:  PV  EXISTING  ASSETS  REPL 

0.0 

0.0 

0.0 

LESS:  PV  INVESTMENT  END  VALUE 

0.0 

0.0 

0.0 

TOTAL  PV  OF  NEW  INVESTMENT  (-) 

11  ,977.1  - 

77.8 

+ 

12,054.9  - 

PV  NET  RECURRING  CST(-)/SAV(+) 

47,704.8 

40,083.8 

7,621.0 

PLUS:  PV  REFURB/MODIF  SAVED 

0.0 

0.0 

0.0 

TOTAL  PV  OF  CST(-)/SAV(+) 

35,727.7 

40,167.6 

4,433.9 

DISCOUNT  RATE  (PERCENT) 

10 

10 

10 

SAVINGS  INVESTMENT  RATIO 

3.98 

*** 

NEG 

RATE  OF  RETURN  ON  INVESTMENT 

75.0 

★** 

NEG 

UNIFORM  ANNUAL  CST(-)/SAV(+) 

(NC) 

(NC) 

(NC) 

***Not  Calculatable 
NC  - Not  Calculated 


FIGURE  V-26.  ALTERNATIVES  ANALYSIS  REPORT 
(3-D  PROCEDURAL  TRAINER  AND  ALTERNATIVE  FROM  TASK  2) 
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EXHIBIT:  FINANCIAL  ANALYSIS  (Continued)... 


• From  the  foregoing,  it  is  seen  that  there  is  a negative  incremen- 
tal return  on  investment  between  the  proposed  3-D  Trainer  and  the 
alternative  identified  in  Task  2. 

• Since  the  alternative  was  not  subjected  to  the  analyses  from  Tasks 
3,  4,  5,  and  6,  these  should  be  performed  to  determine  if  it  should 
be  accepted  without  the  investment  of  risk  reduction  project  funds. 

If  substantial  risk  reduction  project  funds  are  identified,  the 
financial  analysis  may  have  to  be  redone. 

PERFORM  SENSITIVITY  ANALYSIS  (07.04) 

Review  Sensitivity  Profile  for  Each  Model 

This  is  similar  to  the  analysis  in  06.07,  however,  it  would  be  restricted 
to  the  tangible  factors  measurable  in  real  dollars.  The  decision  variable 
being  tested  in  this  analysis  is  the  return  on  investment  factor. 

Identify  Sensitive  Variables  to  be  Analyzed 

Variable  Confidence  Range 

OJT  Time  1.5  to  .5  weeks 

Attrition  3.5%  to  1% 

(An  output  report  as  shown  in  06.07  procedures  was  not 
generated  since  none  of  the  variables  caused  a change 
in  the  decision  across  the  confidence  range.) 

TASK  8 - MAKE  ACCEPT/REJECT/STUDY  DECISION.  The  major  outputs  of  this  task 
will  be  left  to  the  reader's  interpretation,  using  the  guidance  outlined  in 
Task  8.  The  presentation  content,  however,  will  include: 

1.  A statement  of  the  problem. 

2.  Background  data. 

3.  The  objective. 

4.  The  proposed  approach  for  meeting  the  objective. 

5.  Supporting  rationale. 

• Assumptions  made. 

• Risks  (Including  projects  to  bring  them  under  control). 

• Benefits/liabilities  (including  savings  and  costs). 
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• Results  of  decision  tree  analysis. 

• Results  of  financial  analysis. 

• Plan  for  accomplishment. 

6.  Summary  conclusions  and  recommendations. 

SCENARIO  NUMBER  TWO  - COST  ANALYSIS  AND  MEASUREMENTS  SYSTEM 

This  scenario  deals  with  a subset  of  the  analytic  and  decision  process  and 
points  out  some  the  considerations  in  the  treatment  of  variable  estimates, 
costs,  risk,  financial  return,  and  measurement  statistics  used  by  training 
management. 

FINANCIAL  ANALYSIS.  An  identified  segment  of  training  presently  handles  10,000 
Student  Days  (DE)  per  year,'  at  a total  cost  of  $1  million.  The  cost  per  SD 
is  $100.  The  $1  million  consists  of  $100,000  in  fixed  costs  and  $900,000  in 
variable  costs.  The  total  cost  of  training  can,  therefore,  be  expressed: 

TOTAL  COST  - $100,000  + $90  X STUDENT  DAYS 

A highly  innovative  proposal  is  submitted  which  permits  students  to  be  trained 
at  twice  the  existing  rate;  however,  it  will  require  an  additional  $200,000 
in  annual  fixed  costs.  The  proposed  training  system  will  have  5000  SD's.  The 
two  training  systems  can  be  described  as  such: 

A (EXISTING)  B (PROPOSED) 


ANNUAL  STUDENT  DEMAND 

500 

500 

AVE  COURSE  LENGTH  (DAYS) 

20 

10 

NO.  OF  STUDENT  DAYS  (SD'S) 

10,000 

5,000 

TOTAL  FIXED  COSTS  ($) 

$100,000 

$300 ,000 

TOTAL  VARIABLE  COSTS  ($) 

$900,000 

$450 ,000 

TOTAL  COST/SD  ($) 

$ 100 

$ 150 

VARIABLE  COST/SD  ($) 

$ 90 

$ 90 

COST/GRADUATE  ($) 

$ 2 ,000 

$ 1 ,500 

The  proposal  obviously  sounds  enticing,  therefore,  the  cost  analysis  should 
provide  management  with  the  appropriate  decision-making  structure  with  which 
to  judge  the  proposal.  The  key  questions  are  "What  are  the  relevant  costs?" 
and  "How  does  the  acceptance  of  the  proposal  affect  the  measurements  which 
managers  at  all  levels  may  be  viewing?" 

Before  addressing  these  questions  specifically,  it  might  be  well  to  look  at  an 
economic  analysis  of  the  proposal.  Assume  that  the  proposal  is  to  be  analyzed 


V-53 


TAEG  REPORT  NO.  12-3 


over  a period  of  five  years  and  that  annual  student  demand  will  be  maintained 
at  500.  An  incremental  analysis  of  the  proposal  gives  the  following  results. 


YEAR 

ANNUAL  SAVINGS 

- ANNUAL  COST 

= NET 

DISCOUNTED 

1 

450 ,000 

- 200,000 

= 250,000 

238,500 

2 

450 ,000 

- 200 ,000 

= 250,000 

216,750 

3 

450,000 

- 200 ,000 

= 250,000 

197,000 

4 

450 ,000 

- 200 ,000 

= 250,000 

179,250 

5 

450 ,000 

- 200,000 

= 250,000 

163,000 

NET  PRESENT 

VALUE 

994,500 

If  the  opportunity  cost  of  capital  were  considered  to  be  10%,  then  one  would 
pay  nearly  $1  million  for  such  an  innovation  in  training.  Even  in  a situation 
where  there  is  some  risk  that  the  annual  demand  will  not  be  sustained,  and  it 


is  decided 
might  look 

to  weight  the  outyears 
as  follows. 

according 

to  the  risk  potential 

, the  analysis 

YEAR  PROJ  1 

DEMAND  RISK  ADJ  DEMAND 

SAVINGS  - 

COST 

= 

NET  DISCOUNTED  @ 10% 

1 500 

X 

.90  = 450 

405,000  - 

200 ,000 

= 

205,000 

195,570 

2 500 

X 

.80  = 400 

360 ,000  - 

200,000 

= 

160,000 

138,720 

3 500 

X 

.70  = 350 

315,000  - 

200,000 

= 

115,000 

90,620 

4 500 

X 

.60  = 300 

270 ,000  - 

200,000 

= 

70,000 

50,190 

5 500 

X 

.50  = 250 

225,000  - 

200,000 

= 

25,000 

16,300 

NET  PRESENT  VALUE  491 ,400 


Here,  with  the  risks  as  stated,  one  might  pay  nearly  $500,000  for  the  innovation. 
Additional  risk  could  be  associated  with  the  training  length.  For  example, 
assume  that  the  probabilities  of  achieving  the  projected  training  length  reduc- 
tions were  as  follows.  (Remember  the  existing  training  length  is  20  days  while 
the  proposed  length,  with  the  innovation,  is  10  days.) 


COURSE  LENGTH 

PROB.  OF  ACHIEVING 

EFF  REDUCTION 

20 

0 

0 

19 

0 

0 

18 

0 

0 

Eff  Course 

17 

0 

0 

Length  = 

16 

0 

0 

10.95  Days 

15 

0 

0 

14 

.05 

.70 

Savings  in 

13 

.10 

1.30 

Course  Length  = 

12 

.15 

1.80 

9.05  Days 
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NEW  COURSE  LENGTH  PROB.  OF  ACHIEVING 


EFF  REDUCTION 


(Continued) 


11 

10 

9 


.20 
.45 
* .05 


2.20 

4.50 

.45 


10.95 


If  this  dimension  of  risk  is  added  to  the  previous  analysis , rather  than  the 
adjusted  demand  being  multiplied  by  a 10-day  savings  per  student,  it  will  only 
be  multiplied  by  9.05.  That  analysis  would  give  the  following  results. 


TOTAL 

DAYS  COST/  DISC  @ 

cfl\/rn  v cn  = cax/tncc  _ rncT  = wtt  in? 


YR  ADJ 

DEMAND 

X 

DAYS  SAVED  = SAVED 

X 

SD 

= 

SAVINGS 

- COST 

= 

NET 

10% 

1 

450 

X 

9.05  = 4072.5 

X 

90 

= 

366,525 

- 200,000 

= 

166,525 

158, 8F5 

2 

400 

X 

9.05  = 3620 

X 

90 

s 

325,800 

- 200,000 

= 

125,800 

109,069 

3 

350 

X 

9.05  = 3167.5 

X 

90 

« 

= 

285,275 

- 200,000 

= 

85,275 

67,197 

4 

300 

X 

9.05  = 2715 

X 

90 

s 

244,350 

- 200,000 

= 

44,350 

31 ,799 

5 

250 

X 

9.05  = 2262.5 

X 

90 

s 

203,625 

- 200,000 

s 

3,625 

2,364 

NET  PRESENT  VALUE  369,294 

The  value  of  the  proposal  is  down  to  $370K,  however,  it  still  appears  attractive. 


From  the  preceding  examples,  it  can  be  seen  that  the  relevant  costs  (or  costs 
avoided)  are: 

1.  The  additional  fixed  cost  ($200,000). 

2.  The  variable  costs  saved  ($90  per  Student  Day). 

These  costs  were  adjusted  in  several  ways  to  account  for  risk,  and  the  net 
savings  were  discounted  at  a 10%  rate  to  account  for  the  time  value  of  money. 
An  additional  adjustment  could  have  been  made  on  the  annual  fixed  cost  if  it 
were  felt  that  some  risk  was  attached  to  that  value  remaining  at  the  speci- 
fied level . 

MEASUREMENTS  ANALYSIS.  An  equivalent  concern  with  adoption  of  a proposal  such 
as  this  relates  to  the  effects  it  might  have  on  the  values  reported  to  manage- 
ment for  measuring  and  controlling  the  training  system.  The  use  of  ratios  is 
quite  common,  such  as  cost/student  hour,  cost/student  day,  cost/graduate,  etc. 
The  introduction  of  the  proposed  innovation  will  affect  certain  ratios  of 
these  types,  some  favorably,  others  adversely.  For  example,  the  proposed 
innovation  will  cause  a 50%  Increase  in  student  day  costs,  while  it  will  cause 
a 25%  decrease  in  the  cost  per  graduate.  The  following  comparison  shows  how 
these  changes  result.  Variables  names  have  been  shortened  to  facilitate 
mathematical  handling. 
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VARIABLE  NAME 

SYMBOL 

• 

Annual  Student  Demand 

GRAD 

Ave.  Course  Length  (Days) 

TLENGH 

• 

No.  of  Student  Days 

SD 

Total  Fixed  Costs 

CFX 

• 

Total  Variable  Costs 

CVRB 

• 

Total  Cost/SD 

CSDT 

Variable  Cost/SD 

CSDV 

• 

Cost/Graduate 

CGRD 

Total  Cost 

CTOT 

• 

VARIABLE  RELATIONSHIPS 

CSDT  * (CFX  + CVRB)/SD 

9 

= (CFX  + CSDV*TLENGH*GRAD)/(TLENGH*GRAD) 

9 

CGRD  = (CFX  + CVRB)/GRAD 

For  A (Existing): 

• 

CSDTa=  ($100,000.  + $90.  * 20  * 500)/(20  * 500) 

= ($100,000  + $900,000)/10,000 

9 

= $1,000,000/10,000  A 

= $100  (COST  PER  STUDENT  DAY)  W 

CGRDa*  ($100,000.  + $900,000)/500 

• 

= $1,000,000/500 

= $2,000  (COST  PER  GRADUATE) 

m 

For  B (Proposed): 

CSDTB»  ($300,000.  + $90.  * 10  * 500)/(10  * 500)  ^ 

= $300,000.  + $450,000.)/5000 

• 

= $750,000./5000 

= $150.  (COST  PER  STUDENT  DAY) 

• 
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CGRDb=  ($300,000.  + $450,000.)/500 
= $750,000./500 
= $1500.  (COST  PER  GRADUATE) 


A failure  to  consider  the  economic  justification  as  well  as  measurement  im- 
pacts, may  lead  to  future  misinterpretations  of  the  value  of  the  innovation, 
once  it  has  been  implemented.  A management  measurement  and  control  system 
which  focused  only  on  Cost  per  Student  Day  (CSDT)  might  be  shocked  at  the 
50%  increase  and  may  not  give  proper  credit  for  adoption  of  an  economically 
sound  innovation. 

Another  important  management  consideration  is  with  the  sensitivity  of  various 
measurements  to  student  loads  under  existing  and  proposed  conditions.  The 
two  previously  considered  measurements,  Cost  per  Student  Day  and  Cost  per 
Graduate,  should  be  analyzed  for  their  profiles  of  variance  under  the  two 
training  situations,  (A)  existing  system  and  (B)  proposed  system. 

In  Figure  V-27,  Total  Costs  and  Cost/SD  are  plotted  for  the  two  alternatives. 

It  can  be  seen  that  under  the  proposed  system  (B),  student  day  costs  are 
higher  for  any  number  of  student  days,  and  they  rise  more  rapidly  as  student 
days  decrease.  However,  a more  relevant  consideration  is  the  break-even  point 
for  Cost  per  Graduate  as  shown  in  Figure  V-28.  If  the  number  of  graduates 
drops  from  500  to  222,  then  the  cost  of  operation  under  the  existing  system 
and  the  proposed  system  is  the  same  ($500,000);  therefore,  the  Cost/Graduate 
is  the  same  ($2222.).  At  the  break-even  point,  however,  the  Cost/SD  under 
the  proposed  system  will  be  twice  that  of  the  existing  system  ($225.  vs.  $112.5). 
This  is  not  a relevant  concern.  What  is  of  concern  is  the  higher  operating 
cost  below  the  break-even  point  of  222  graduates  under  the  proposed  system. 

This  is  due  to  the  higher  leverage  provided  by  the  increased  level  costs.  A 
higher  Cost/Graduate  exists,  also,  under  the  proposed  system,  as  the  number  of 
graduates  falls  below  the  break-even  point. 

There  are  several  alternatives  for  maintaining  the  continuity  of  the  previous 
measurements  if  the  proposal  is  accepted.  One  is  to  use  the  Cost/SD  (B)  as  a 
control  value  for  all  subsequent  numbers  of  graduates.  Even  though  it  is 
higher,  future  cost  problems  can  be  detected  from  a higher  Cost/SD  than  pro- 
jected at  the  time  of  the  analysis.  Another  alternative  is  to  adjust  all 
future  student  day  costs  by  a ratio  of  the  before- to-af ter  costs  at  the  time 
of  the  analysis.  Since  this  value  will  vary  depending  on  the  number  of 
graduates,  it  must  be  calculated  for  each  level.  The  procedure  would  be  as 
follows: 

Let  CSDTg  be  the  Cost/SD  measured  and  observed  at  some  future  date. 

Let  CSDTm  be  the  Cost/SD  adjusted  to  the  preproposal  base  and  the  value 

that  will  be  presented  to  management. 


Then : 


CSDTm  = CSDTg  X CSDTa 
E5T5Tb 
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FIGURE  V-27.  TOTAL  COST/COST  PER  STUDENT  DAY  BREAK-EVEN  ANALYSIS 
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FIGURE  V-28.  TOTAL  COST/COST  PER  GRADUATE  BREAK-EVEN  ANALYSIS 
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Since  CSDL\  and  CSDTg  are  calculable  as  a function  of  numbers  of  graduates  (GRAD) 
at  time  of  proposal  analysis,  the  equation  reduces  to; 

CSDTm  = CSDTn  X 1000  + 18  GRAD 
M 0 6000  + 18  GRAD 

For  example,  assume 

CSDT0  = $160 

GRAD  - 500 

Then: 

CSDT„  = $160  X 1000  + 9000 
6000  + 9000 

« $160  X .67 

= $107. 

Since  at  a level  of  500  graduates  prior  to  the  proposal,  management  would  have 
expected  the  Cost/SD  to  be  $100,  the  CSDT»  = $107,  reflects  a cost  problem 
which  should  be  investigated  further. 

SUMMARY.  The  purpose  of  this  scenario  has  been  to  unfold  several  of  the  more 
important  factors  in  evaluating  an  innovation  proposal;  namely,  the  identifica- 
tion of  relevant  variables,  the  economic  justification  process,  and  the  effects 
upon  the  measurements  and  control  system. 

SCENARIO  NUMBER  THREE  - SHIPBOARD  TRAINING  WITH  MICROFICHE 

The  following  is  an  example  of  the  use  of  ETAM  in  assessing  the  potential  of 
using  microfiche  as  a medium  with  a simple  M/F  viewer  for  shipboard  training 
of  courses  or  course  content  now  taught  on  shore-based  installations. 

In  this  scenario,  it  is  understood  that  the  assessment  operations  are  to  be 
applied  principally  to  Tasks  1,  3,  and  5 in  ETAM.  Cost  data  are  not  to  be 
used  and  computed,  but  the  general  classes  of  cost  information  should  be  iden- 
tified. The  major  point  in  this  exercise,  however.  Is  the  use  of  Task  5, 
Range-of-Effect,  structure  on  this  problem. 

The  writer,  who  served  as  analyst  here,  has  limited  background  in  the  use  of 
microfiche  for  training  purposes.  He  was  given  several  study  reports  into  M/F 
use,  some  briefing  by  Dr.  Braby  of  TAEG,  and  some  literature  on  extensions  of 
microimage  technology  into  training,  such  as  the  Lincoln  Lab  technology.  This 
was  the  analyst's  background  to  the  present  assessment  results. 

Since  an  assessment  must  have,  at  least  as  a starting  point,  a fixed  reference 
as  to  the  entity  being  assessed,  it  was  decided  to  use  the  standard  microfiche 
and  the  simple,  portable  viewer  as  the  "innovation."  If  instructional  or  other 
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functions  were  to  be  added  to  this  "system"  (such  as  the  elaborate  Lincoln  sys- 
tem contains),  these  additions  would  and  could  be  superimposed  on  the  present 
pattern  of  benefits  and  costs  as  modifications  or  extensions. 

Because  this  report  is  intended  to  have  instructional  value  on  ETAM  as  well  as 
being  a trial  operation,  the  text  is  fairly  extended.  Many  of  the  reasons  for 
making  the  stipulations  in  the  assessment  that  appear  here  have  been  reported, 
perhaps  at  more  than  necessary  length.  Issues  have  been  raised  and  explored 
which,  In  the  present  problem,  have  led  into  a dead  end.  It  is  believed  that  a 
full  exposition  of  this  kind  enables  others,  as  contributors  to  the  assessment, 
to  work  with  the  rationale  for  a conclusion,  as  well  as  with  the  conclusion  it- 
self. Thus,  it  would  be  surprising  if  there  were  not  some  strong  disagreements 
with  some  of  the  opinions  asserted  as  "assessments."  Clarification  can  be  pro- 
gressive, however,  when  the  participants  to  a controversy  can  sort  out  whether 
their  disagreement  Is  based  on  assumptions,  stipulations,  rationales,  or  infer- 
ences and  judgments. 

The  preparation  of  this  entire  assessment,  including  more  than  27  pages  of  text, 
has  occupied  between  four  to  five  days  of  time.  The  setting  down  of  the  actual 
assessment  data  In  the  report  --  without  the  rationales,  etc.  --  with  the  infor- 
mation at  hand,  would  have  taken  between  one  and  two  days,  and  closer  to  one 
than  to  two.  These  estimates  are  intended  to  suggest  that  it  does  not  neces- 
sarily take  forever  to  go  through  the  range-of-effect  structure.  As  a matter 
of  fact,  describing  the  innovation  in  Task  1 would  have  taken  more  than  half  of 
the  time  given  to  the  assessment  operation  through  Task  5. 

EXHIBIT:  ETAM  EVALUATION  OF  MICROIMAGE  TECHNOLOGY  FOR  TRAINING 

TASK  1 - FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION 

MICROFICHE  TECHNOLOGY 

Objectives  and  Summary  (01.01) 

The  general  proposal  is  aimed  at  the  substitution  of  a system  for 
producing  and  using  micro-images,  such  as  microfiche,  for  paper  as 
a medium  for  individualized  instruction  on  shipboard.  The  microfiche 
type  of  card  can  be  coupled  with  a viewing  mechanism  that  enables 
student  responses  to  be  evaluated.  The  result  of  the  evaluation 
can  be  the  basis  for  the  selection  of  what  is  next  presented  to  the 
student.  It  is  also  feasible  to  develop  techniques  which  simplify 
the  creation  and  modification  of  instructional  content  by  authors, 
using  a computer  to  assist  in  formatting,  indexing,  sequencing,  and 
retrieving  instructional  material.  The  general  benefits  made  possible 
by  this  technology  are: 

1.  Reduction  of  storage  space  for  equivalent  content  on  paper, 
from  50  to  1 , with  a potential  of  200  to  1.  About  the  same 
proportional  reduction  in  weight  for  transport  and  distribution 
purposes . 

2.  Somewhat  greater  flexibility  in  the  amount  and  range  of  content, 

and  In  content  sequencing:  this  enables  options  in  sequencing 

that  Increase  Instructional  capability. 
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3.  Substantial  reductions  in  costs  of  updating  and  reproducing  of 
images  as  compared  with  printing  on  paper.  These  reductions  in 
cost  can  be  increased  by  computerizing  the  means  whereby  sets 
of  images  are  created;  layouts  are  generated  by  the  author;  and 
indexing,  retrieval,  and  collation  of  images  is  managed. 

4.  The  technical  feasibility  for  shifting  a large  burden  of  instruc- 
tion from  shore  schools  to  individualized  shipboard  instruction 
in  a substantial  range  of  knowledge  and  in  some  skills. 

Because  of  the  substantial  development  and  operating  costs  of  the 
proposed  technology,  the  payoff  must  come  from  a large  scale  of 
application.  Furthermore,  any  Implementation  should  enable  exten- 
sions of  other  technologies,  such  as  update  by  satellite  communica- 
tion, or  conversational  interaction  between  the  student  at  a micro- 
image terminal  with  a central  computer,  to  be  readily  integrated  to 
existing  facilities. 

Delivery  System  and  Training  Vehicle  (01.02) 

Microimages  are  contained  on  transparent  cards.  One  of  the  standard 
sizes  Is  4"  x 6".  Between  60  to  several  hundred  images  are  on  a card. 
Greater  reductions  are  possible.  The  card  is  inserted  into  an  optical 
viewer  which  enlarges  the  image  to  readable  size.  Substantial  advances 
have  been  made  in  the  quality  of  viewers,  and  in  compactness  of  the  de- 
vice. The  desired  Image  Is  located  manually;  movement  from  one  image 
to  the  next  is  manual.  There  are  various  forms  of  indexing  assistance 
for  locating  an  image  and  mechanical  devices  for  advancing  image  frames. 
Manual  advance  seems,  until  now,  as  effective  as  mechanical  aids.  The 
cost  of  automatic  sequencers  would  greatly  increase  the  cost  of  the 
device  and  increase  failure  rate  and  maintenance  costs. 

The  proposal  does  not  include  a firm  specification  for  the  properties 
of  the  viewing  device.  A reference  base,  however,  for  further  cost- 
benefit  analysis  and  comparison  is  the  simple,  portable  viewer  which 
projects  the  equivalent  of  an  8-1/2"  x 11"  display  with  a resolution 
of  150  lines  to  the  inch.  The  microfiche  card  is  manually  inserted 
and  Images  are  manually  selected  and  advanced. 

At  present,  it  is  assumed  that  the  formal  testing  of  what  the  student 
learns  at  the  level  of  specific  course  objectives  would  be  made  on 
another  medium  and  device,  perhaps  served  by  a human  instructor. 

Jobs  and  Tasks  Applicable  (01.03) 

Preliminary  examination  suggests  that  anything  that  can  be  learned 
from  print  on  paper  can  be  learned  by  microimage.  This  applies  to 
the  large  body  of  knowledge  material  called  "Knowledge  Factors"  in 
occupational  standards.  To  the  extent  that  a knowledge  factor  is 
necessary  to  the  performance  of  a skill  (or  "Practical  Factor"),  this 
medium  is  applicable.  Most  procedural  tasks  require  a knowledge  substrate. 
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On  this  premise,  practically  all  jobs  and  job  advancement  opportunities 
would  be  candidates  for  this  facility. 

Relevant  Student  Attributes  (01 .04) 

The  requirement  is  a reading  ability  at  the  level  of  programmed  in- 
struction for  the  given  instructional  topic  and  training  objective. 

The  student  may  require  somewhat  higher  levels  of  self-discipline 
than  would  be  needed  in  formal  school  where  time  is  structured  and 
study  activities  are  more  or  less  supervised.  Another  complication 
could  be  the  fact  that  on  shipboard,  the  student  might  feel  his  pri- 
mary responsibility  was  his  job  — and  properly  so  — but  this  moti- 
vation could  lead  to  rationalizations  that  the  effort  required  for 
self-instruction  could  be  postponed.  For  many  students,  some  exter- 
nally applied  incentive  might  need  to  be  applied  on  a daily  or  weekly 
basis. 

The  lack  of  the  stress  of  schedules  whereby  schools  must  operate 
classes,  however,  would  enable  slower  students  to  spend  more  time 
in  mastering  subject  matter.  The  consequence  could  be  a lower  wash- 
out rate,  at  least  on  the  basis  of  applied  ability  level. 

Empirical  Data  on  Benefits  and  Savings  (01.05) 

Samples  of  data,  both  on  the  potential  liabilities  and  benefits  in 
learning  with  the  microfiche  system  and  its  costs,  are  in  process  of 
being  collected. 

At  least  indirect  evidence  shows  that  microfiche  is  at  least  as  good 
as  paper  manuals,  texts,  and  programmed  instruction. 

The  physical  density  of  storage  of  content  on  microfiche  makes  it 
feasible  to  make  a large  variety  of  instructional  material  available 
to  the  learner.  This  enables  him  to  satisfy  his  curiosity,  and  may 
enable  additional  learning  through  redundancy:  the  same  idea  or 
pattern  of  ideas  may  become  not  only  better  learned  and  understood 
(paraphrased  in  various  contexts)  but  better  remembered  and  potentially 
better  applied  to  appropriate  occasions. 

The  negative  possibility  is  that  most  students  will  tend  to  learn 
material  only  to  a minimum  level  of  grasp  if  they  are  left  to  set 
their  own  criterion.  This  means  rapid  forgetting,  even  if  the  test 
is  passed.  An  external  means  of  controlling  amount  of  practice  and 
criterion  level  reduces  this  liability.  However,  this  liability 
exists  in  printed  materials. 

Practical  Importance  of  the  Training  Product  (01.06) 

A large  majority  of  skills  has  a substantial  knowledge  component, 
either  as  background  for  acquiring  the  skill,  or  as  operational  and 
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reference  information  for  exercising  the  skill.  To  the  extent  that 
opportunity  and  incentive  can  be  created  on  shipboard  for  acquiring 
these  knowledges,  a double  savings  can  be  effected.  One  is  the  re- 
duction in  classes  and  class  time  in  shore  installations.  A second 
is  the  reduction  in  the  proportion  of  enlistment  time  during  which 
the  incumbent  is  not  in  operational  status  — at  least  following  his 
first  enlistment's  period  of  schooling. 

If  this  innovation  can  be  realized,  the  achievement  of  its  goals  have 
substantial  practical  importance. 

Resource  Requirements  for  Further  Study  and  Implementation  (01.07) 

A program  has  not  yet  been  prepared. 

Although  the  right  hardware  may  not  be  available,  no  break-through  in 
technology  is  required  for  microfiche.  The  anticipated  use  of  holo- 
graphies which  could  extend  the  instructional  possibilities  perhaps 
substantially  would  require  some  technological  break-throughs  both 
in  cost  and  in  miniaturization  of  components. 

It  appears  that  considerably  more  effort  may  need  to  be  spent  in 
determining  ways  of  using  the  medium  and  its  possibilities  for  instruc- 
tion. Some  useful  principles  have  been  invented  and  applied,  but  with 
limited  empirical  data. 

Although  the  unit  cost  of  the  microfiche  viewer  is  relatively  small, 
it  is  not  trivial.  But  the  development  of  a coordinated  system  for 
generating  and  administering  large  scale  curricula  would  require  a 
mul ti -mi  1 1 i on  dollar  effort. 

A somewhat  higher  level  of  average  talent  is  likely  to  be  needed  to 
develop  effective  microfiche  instructional  content  than  is  character- 
istic of  classroom  instructors  and  of  programmed  instruction  designers. 
The  latter  have  the  instructional  machine  as  a surrogate  human  instruc- 
tor to  keep  the  student  going  through  to  the  end  of  an  instructional 
segment.  The  student  with  book  or  microfiche  can  more  readily  turn 
off  without  a sense  of  nonccmpletion  of  a responsibility. 

TASK  2 - DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION.  Instructional  material 
on  paper  is  the  primary  alternative.  The  microimage  may  be  cost-justified  over 
paper,  quite  Independent  of  other  benefits. 

There  me*y  be  technical  alternatives  to  microfiche  and  the  simple  microfiche 
viewer.  It  is  unlikely  that  they  will  be  lower  cost  per  frame  of  displayed 
information,  but  they  may  turn  out  to  be  lower  overall  cost  per  unit  of  learned 
product  such  as  a knowledge  objective  or  a skill  objective,  and  across  a wider 
range  of  subject  matter  and  learning  levels. 

Those  comparisons  are  not  a part  of  the  present  evaluation,  except  insofar  as 
this  evaluation  provides  a reference  for  comparisons. 
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TASK  3 - PRELIMINARY  FEASIBILITY  PROFILE 

Importance  of  the  Innovation  to  the  Projected  Navy  Mission  (03.01) 

Shipboard  training  is  an  alternative  to  shore  training,  and  therefore 
cannot  be  considered  crucial.  In  the  event  of  an  extended  military 
emergency  and  full-scale  mobilization,  the  ability  to  upgrade  the  capa- 
bilities of  every  person,  wherever  he  may  be,  and  to  do  so  rapidly, 
could  be  crucial.  But  this  is  not  the  case,  nor  is  it  a present 
planning  consideration. 

Secondly,  the  training  of  knowledges  is  subordinate  to  training  in 
actual  job  skills  as  such  where,  in  most  cases,  the  learning  and 
application  of  the  knowledge  is  likely  to  be  most  effective  and 
efficient. 

Taking  both  of  these  arguments  into  account,  the  estimate  would  have 
to  read: 

Shipboard  training  with  microfiche  and  what  can  be  learned 
from  microfiche  technology  would  have  an  effect  of  moderate 
importance. 

But  taking  into  account  that  relatively  few  pounds  of  microimages 
as  training  material  and  as  job  aids  (technical  manuals)  can  dis- 
place tons  of  paper  containing  equivalent  information,  the  picture 
changes  rather  radically: 

The  effect  can  be  substantial  and  important. 

Organizational  Compatability  (03.02) 

Shipboard  training,  as  contrasted  with  shore-based  training,  is 
irrelevant  to  Navy  command  and  career  structure,  or  supportive  of 
career  structure.  Since  the  policies  of  the  Navy  are  to  support  the 
fullest  possible  growth  potential  of  all  of  its  personnel,  shipboard 
training  is  indeed  supportive  and  relevant. 


To  the  extent  that  an  organization  prefers  to  maintain  complete  con- 
trol of  both  the  process  and  the  product  it  is  charged  with  producing, 
it  is  understandable  that  the  training  institutions  would  prefer  full- 
time students  studying  in  the  training  establishment  itself.  The 
student  within  this  jurisdiction  has  direct  accountability  only  to 
the  training  officers  and  management,  and  thus  the  training  manage- 
ment has  the  authority  to  discharge  its  responsibility  for  producing 
trained  men. 
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On  the  other  hand,  training  management  can  perceive  its  objective  as 
achieving  training  at  lowest  cost  and  penalty  to  the  Navy  as  an  oper- 
ational enterprise.  In  this  latter  light,  shipboard  training  is  an 
extension  of  the  training  enterprise  with  a concern  for  cost  as  well 
as  the  benefit  of  the  product. 

A realistic  assessment  would  be  that  some  minor  conflicts  of  a temp- 
orary nature  would  arise  within  the  training  establishment,  not  on 
the  microfiche  program  as  such,  but  against  shipboard  training  on  a 
wide  and  formalized  scale. 

Estimate  of  State-of-the-Art  Applicable  to  the  Innovation  (03.04) 

With  respect  to  the  technical  effectiveness  of  instruction  with  more 
or  less  existing  microfiche  technology,  it  seems  realistic  to  expect 
a high  probability  of  success  with  at  least  a moderate  R&D  effort. 

This  effort  would  be  most  profitably  spent  in  the  techniques  of 
successful  instruction  with  this  medium,  and  in  techniques  of  gener- 
ating or  organizing  such  instruction. 

Estimate  Further  R&D  Funding  (03.05) 

Programs  for  R&D  in  this  enterprise  have  not  been  formulated.  But  a 
full-scale  implementation  across  a wide  variety  of  courses,  job  skills 
and  knowledges,  and  students  would  require  a moderate  level  of  funding 
somewhat  above  the  level  of  present  planned  and  integrated  programs. 

It  would  be  possible  to  demonstrate  feasibility  on  a relatively  re- 
stricted scale  v/ith  relatively  modest  funds.  Such  a demonstration 
would  lack,  however,  any  basis  for  assessment  of  the  economic  benefit 
and  the  instructional  range  and  flexibility  of  a full-scale  develop- 
ment which  included  computerized  techniques  for  preparing  and  dis- 
seminating training  content. 

Required  Technical  Support  (03.06) 

The  kinds  of  talent  that  would  be  needed  for  developing  these  training 
materials,  and  for  administering  this  entire  instructional  program, 
mlqht  be  unusual  but  not  rare.  Adequate  talent  could  be  made  avail- 
able. 

Attltudlnal  Acceptance  (03.07) 

In  the  case  of  shipboard  training,  the  key  requirement  for  attitudinal 
acceptance  will  be  that  of  the  conmanding  officer  of  the  ship.  Very 
reasonably,  he  feels  his  primary  mission  and  responsibility  is  the 
effective  operation  of  the  ship.  He  is  unlikely  to  feel  that  the 
training  of  personnel  on  his  ship  is  a legitimate  competitor  to  the 
operational  mission.  If  shipboard  inadequacies  are  traced  to  this 
"excuse",  the  commander  Is  likely  to  downgrade  the  training  mission. 
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This  set  of  values  will  quickly  be  transmitted  to  trainees.  Since 
self-instruction  requires  considerable  incentive,  the  loss  of  the 
drive  in  this  incentive  will  impair  motivation  to  take  the  training 
seriously.  Thus,  both  opportunity  and  incentive  can  be  largely  de- 
stroyed. 

It  is  unrealistic  to  expect  that  commanders  brought  up  in  traditional 
Navy  doctrine  will  accept  a tradeoff  criterion  between  any  threat  to 
shipboard  effectiveness  and  training  objectives  that  do  not  point 
directly  to  an  increase  or  at  least  maintenance  of  shipboard  effective- 
ness. The  acquisition  of  new  knowledge  and  skills  for  promotion  to 
another  grade  does  not  point  directly  to  such  an  increase  in  the  capa- 
bility of  the  ongoing  mission  of  the  ship. 

This  is  a major  impediment,  not  to  microfiche  as  such,  but  to  the 
instructional  environment  of  its  intended  use. 

The  assessment  must  be:  The  community  in  which  the  innovation  is  to 

be  introduced  will  respond  with  sustained  hostility. 

Many  potential  students  on  ship  may,  even  with  the  encouragement  of 
the  commander  for  training,  be  ambivalent  about  time  and  effort  spent 
in  training  that  is  away  from  their  duties  and  comrades,  or  from  re- 
laxation. 

TASK  5 - DETERMINE  RANGE-OF-EFFECT.  Some  preliminary  comments  are  in  order 
before  plunging  into  the  procedural  operations  of  Task  5.  Clearly,  one  pro- 
posed advantage  of  microfiche  is  in  its  various  economies  over  its  equivalent 
in  paper  on  shipboard  --  plus  perhaps  shore-based  training  and  operations.  One 
could,  without  going  further  into  the  specifics  of  range-of-effect,  make  dollar 
comparisons  of  the  M/F  and  paper  alternatives  for  existing  shipboard  materials. 
Substituting  M/F  for  much  of  the  present  shipboard  paper  might  be  justified  on 
this  basis  alone. 

The  second  problem  is  the  justification  for  using  M/F  for  shipboard  training. 
Here,  the  major  question  — assuming  acceptance  of  shipboard  training  by  the 
command  levels  --  is  what  can  be  taught  and  learned  with  the  M/F  medium?  The 
most  comprehensive  answer  here  is  "knowledge  content."  So  before  making  further 
analysis,  a good  tactic  would  be: 

1.  Identify  knowledge  courses  now  in  the  Navy  training  curriculum  that 
could  be  candidates  for  shipboard  training. 

2.  Assuming  at  least  equal  --  and  probably  superior  — training  benefits 
from  M/F  as  from  paper,  determine  the  cost  advantages  of  shipboard  as 
contrasted  with  shore-based  training  on  knowledge  material. 

This  relatively  simple  analysis  could  In  itself  lead  to  a justification  for  M/F 
and  shipboard  training.  Clearly,  this  would  not  constitute  a full  "range-of- 
effect"  analysis,  but  this  may  be  unnecessary  for  reaching  a decision  to  use  it 
rather  than  alternatives. 
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If,  however,  the  decision  makers  demand,  as  a basis  for  their  decision,  some 
range  of  training  in  task-skills  to  be  specified  in  addition  to  knowledge  train- 
ing, then  the  following  steps  will  have  to  be  performed. 

These  steps  might  also  be  performed  if  the  objective  were  not  merely  that  of 
accept  or  reject  M/F  as  a medium  on  the  least  amount  of  information  that  would 
enable  such  a decision,  but  to  make  a research  type  inquiry  into  the  possibil- 
ities of  M/F  for  the  fullest  range  of  potential  applicability.  Such  informa- 
tion would  be  useful  and  perhaps  essential  as  a basis  for  a system  design 
structure  that  used  M/F  or  a similar  medium,  for  instructional  purposes.  It 
should  be  noted  that  although  ETAM  is  intended  to  serve  this  kind  of  purpose  — 
as  demonstrated  on  the  following  pages  of  analysis  --  the  performance  of  this 
purpose  will  entail  more  labor  than  the  minimum  that  might  be  required  for  pro- 
ducing "accept-  reject-study  further"  kinds  of  decisions. 

The  exercise  of  assessment  will  assume  that  one  or  both  of  the  objectives  cited 
in  the  two  preceding  paragraphs  applies  to  the  problem. 

(The  point  should  be  underlined,  however,  that  because  ETAM  is  comprehensive, 
its  application  Is  necessarily  extensive,  no  matter  what  the  problem  of  assess- 
ment may  be.) 

Choose  Major  Routes  for  Ranqe-of-Effect  (05.01) 

It  is  expected  that  part  of  the  justification  for  microfiche  techno- 
logy can  be  developed  from  displaceable  costs  in  the  printed  paper: 
costs  of  paper,  printing,  distribution,  storage,  updating  and  obsol- 
escence. So  one  prong  of  inquiry  will  be  that  of  Instructional 
Vehicles. 

It  is  also  important  to  determine  the  range  of  instructional  applica- 
bility of  the  microfiche  system.  The  objective  is  to  determine  the 
teaching/learning  hours  of  shore-based  training  in  classrooms  across 
all  applicable  course  materials  that  can  be  displaced  by  shipboard 
training  that  can  be  achieved  through  microfiche.  The  analysis  will, 
therefore,  attempt  to  identify  the  types  or  classes  of  knowledge  and 
skill  that  can  be  acquired,  in  whole  or  in  some  specified  part, 
through  microfiche  usage.  The  specification  of  what  instructional 
objectives  are  applicable  to  microfiche  will  then  be  applied  to  ex- 
isting courses  in  order  to  quantify  the  potential  range  of  applica- 
bility of  M/F  on  shipboard. 

At  a later  time,  when  plans  have  been  completed  for  computer-assisted 
development,  distribution  and  management  of  training,  and  job  aid  de- 
velopment with  M/F  technology,  a third  route  of  assessment  will  be 
through  the  Development  and  Administrative  Management  of  Training 
route  in  ETAM.  The  M/F  program  has  not  developed  to  a point  where 
this  analysis  would  now  be  profitable.  It  is  expected  that  the 
following  categories  in  the  D&A  route  would  be  applicable: 
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• Developing  training  requirements  from  skill  requirements 
descriptions. 

• Selection  or  devising  of  training  modules  and  modularities. 

• Designing  the  content  of  instruction. 

Although  microfiche  as  an  instructional  vehicle  does  not  have  proper- 
ties directly  applicable  to  these  D&A  issues,  a semi -computerized 
subsystem  which  produced  and  selected  microimages  can  be  devised 
that  would  enable  this  medium  to  be  more  attractive  than  paper  equiv- 
alents. 

Conwent 

The  present  assessment  will  extend  to  the  use  of  M/F  as  a substitute 
for  paper  job  aids  such  as  technical  manuals  on  shipboard.  The  assess- 
ment will  assume  operational  feasibility  for  the  substitution  of  M/F 
displays  for  paper,  supplemented  with  printer-copiers  of  M/F  images 
where  paper  copies  are  essential  at  a given  time.  The  operational 
justification  for  this  assumption  will  not  enter  into  the  present 
assessment,  which  will,  therefore,  be  limited  to  the  economi c oppor- 
tunity in  substituting  M/F  for  paper. 

Vehicle  Type  That  Fits  the  Innovation  (05.03) 

The  microfiche  and  its  viewer  are  applicable  to  the  full-range  of 
static  graphics  used  in  instruction.  It  can,  thus,  display  any  con- 
tent  shown  on  paper,  foil,  slides,  or  static  images  on  a display 
tube.  Fiche  Images  can  be  used  to  project  enlarged  images  for  group 
discussion  or  classroom  presentation,  thus  displacing  slides  and 
foils. 

Note  that  an  economic  assessment  could  be  made  at  this  point: 

a.  Determine  present  costs  of  all  shipboard  training,  demon- 
stration and  briefing  materials  contained  on  paper,  slides, 
charts,  etc. 

b.  Determine  costs  for  microfiche  equivalents.  Assume  a rea- 
sonable requirement  of  paper  duplication  --  say  10%  of  the 
total.  In  both  cases,  costs  should  include  generating  and 
editing  the  content,  updating  it,  distributing  it,  storing 
it,  and  accessing  it  selectively. 

c.  Compare  the  two  sets  of  costs.  A preliminary  justification 
for  M/F  may  appear  and  be  more  than  marginal.  In  any  event, 
a cost  ratio  between  M/F  and  other  media  with  equivalent 
function  will  be  generated. 
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Type  of  Training  Objective  (05.04) 

In  general  terms,  microfiche  can  be  used  for  the  learning  of: 

a.  Reference  knowledge. 

b.  Knowledge  that  is  task-specific  and  enabling  --  to  the 
extent  that  the  knowledge  can  be  acquired,  retained,  and 
applied  as  a verbalized,  cognitive  content. 

c.  Some  task-skill  formats  at  preliminary  stages  of  compe- 
tence. 

d.  Some  skills  that  are  primarily  symbolic  transactional, 
such  as  filling  out  a requisition  or  a duty  roster. 

These  may  be  treated  as  "task  formats",  however. 

The  foregoing  identifications  mean  that  specifics  will  be  sought  in 
the  assessment  routes  leading  to  Reference  Knowledge,  Enabling  Knowl- 
edge, and  Task  Formats. 

Select  Vehicle  Properties  (05.05) 

At  this  point  we  become  concerned  with  displaceable  costs  in  the  pre- 
sent Inventory  and  usage  of  M/F  alternatives  In  current  training 
courses. 

Since  the  "Instructional  Functions"  in  05.05  are  roughly  equivalent 
between  the  M/F  technology  and  the  alternatives  It  Is  displacing 
(paper,  slides,  etc.),  these  functions  are  not  pertinent  to  this 
assessment  and  can  be  disregarded. 

The  "Utilization  Functions"  that  do  apply  for  comparison  purposes 
consist  of: 

Programmability  of  instructional  content. 

Portability  and  storage. 

Multiple  usage. 

Maintenance. 

Combining  instructional  capabilities  distributed  among  several 
media/devices. 

Modularization  of  training  content  for  more  general  applicability. 

Cost  advantages  and  liabilities  may  arise  in  each  of  these  categories 
of  cost  assessment.  It  should  be  recognized  that  raw  dollar  estimates 
of  saving  may  present  an  Incomplete  picture  of  value.  For  example, 
it  Is  common  practice  to  assess  the  cost  of  building  space  on  a per 
square  foot  basis.  But  on  shipboard,  construction  costs  per  square 
foot  of  floor  space  will  be  an  incomplete  evaluation  of  savings.  The 
dramatic  example  Is  cubic  foot  space  in  a submarine.  The  assessor 
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may,  therefore,  want  to  get  the  help  of  appropriate  operations  people 
to  develop  a "functional  multiplier"  for  the  unit  of  objective  cost. 
Thus,  on  a given  type  of  ship,  the  construction  cost  per  square  foot 
may  have  a multiplying  factor  of  3 for  space  saving.  But  because  of 
considerable  idle  time  of  shipboard  electronics  maintenance  men,  the 
normal  "maintenance  cost"  for  simple  instructional  devices  might  well 
be  .5.  Obviously,  when  a multiplier  of  this  kind  is  used,  both  the 
multiplier  (and  its  rationale)  and  the  cost  factor  as  objectively 
determined  should  be  presented  to  the  decision  maker. 

The  portability  of  a compact,  lightweight  M/F  viewer,  plus  the  instruc- 
tional and  reference  context  in  a few  square  inches  of  M/F  cards,  can 
be  assessed  in  two  ways.  One  is  in  the  readiness  with  which  a rela- 
tively small  stock  of  viewers  can  be  borrowed,  used,  and  returned  by 
a number  of  trainees.  There  is,  thus,  high  opportunity  for  much  usage 
per  instructional  entity  or  unit.  The  second  point  of  view  is  that 
the  combination  of  compactness  and  low  cost  makes  it  possible  for  all 
learners  to  have  a copy  of  the  device  and  instructional  materials  in 
their  private  living  quarters  where  it  is  continuously  accessible  to 
the  spare  moments  of  the  learner,  as  well  as  for  more  extended  periods 
of  study  time. 

(Incidentally,  books  may  be  printed  in  M/F,  and  these  may  include 
reading  for  entertainment  as  well  as  study  materials.  This  may  add 
to  M/F  justification.) 

Specifications  for  Searching  the  Instructional  Vehicle  Inventory  (05.06) 

In  this  example,  the  search  specification  is  simple:  it  consists  of 

"static  graphics"  as  defined  in  the  ETAM  context. 

In  this  case,  however,  it  is  simple  to  become  more  specific  in  identi- 
fying the  names  of  the  relevant  media.  These  are: 

All  printed  materials  used  for  Instructional  purposes  except 
student  evaluation  tests. 

All  slides,  foils,  and  other  single  image  transparencies  for 
projection  purposes. 

All  charts,  diagrams,  tables,  pictures. 

Manuals  of  operating  instructions  — if  they  are  used  for  teaching 
purposes.  If  these  manuals  will  be  used  on  the  job  in  M/F  form, 
there  is  a clear  cost  advantage  for  using  them  In  M/F  training. 

If  this  is  not  the  case,  a separate  assessment  should  be  made 
that  compares  a redundant  M/F  version  for  training  with  paper 
manuals  used  both  for  training  and  on  the  job.  It  should  be 
recognized  that  the  shift  from  M/F  to  paper  will  call  for  some 
adjustment  by  the  incumbent,  but  this  may  be  transitory,  espe- 
cially if  there  is  a common  indexing  structure  to  the  two  versions. 
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Identification  of  Relevant  Inventory  of  Training  "Vehicles"  (05.07) 

The  listing  of  types  of  static  image  representations  culled  out  of 
the  total  inventory  list  will  identify  the  contexts  of  application 
and  use.  Of  the  total  list,  a number  of  items  may  be  judged  inappli- 
cable for  M/F  treatment.  For  example,  a class  schedule  which  directs 
where  and  when  a student  will  take  what  classes  should  obviously  be 
on  paper  that  the  student  can  carry  with  him.  Some  condensed  "pocket 
guides"  of  definitions,  general  structures  of  relationships  (such  as 
the  schematic  of  a procedure  or  a device)  should  be  on  paper  carried 
by  the  student  for  rapid  reference  either  at  odd  moments  or  in  con- 
nection with  study. 

After  making  a number  of  detailed  examinations  of  items  in  the  list, 
the  assessor  may  decide,  on  the  basis  of  the  sample  he  has  examined, 
that  a reasonable  estimate  is  that  95%  of  all  the  applicable  types 
of  a given  medium  (say,  paper  print)  is  subject  to  M/F  application 
and  apply  this  formula  to  the  remainder  of  the  relevant  items.  The 
potential  error  in  the  final  assessment  contributed  by  such  estimates 
will  probably  fall  into  the  cracks  of  estimating  unreliability. 

Identify  Benefit  Pattern  for  Instructional  Vehicles  (05.08) 

Separate  comparison  breakdowns  of  unit  costs  would  be  prepared  for 
Instructional  materials  on  paper,  on  slides,  foils,  charts,  and  for 
M/F  for  similar  materials. 

Cost  data  would  be  obtained  from  samples  typical  of  each  medium 
according  to  the  following: 

Preparation  of  the  image  content  to  a process-ready  state. 

Image  reproduction,  singly  and  in  groups  representative  of  a 
collected  unit  — such  as  a booklet,  or  training  element. 

Cost  of  raw  materials;  spoilage  and  wastage  factors. 

Distribution  costs. 

Update  (with  typical  examples)  costs  to  process,  distribute  and 
enter  into  user  image  files. 

Inventory  maintenance  costs:  storage,  handling,  obsolescence. 

Medium  holding  and  handling  materials:  book  binding,  slide 

mounts  and  holders,  etc. 

Associated  display  device  costs  per  unit  of  use;  e.g.,  M/F 
viewer  or  slide  projector. 
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To  these  cost  comparisons  will  be  added  benefits  data  from  Task  Train- 
ing analysis. 

Identification  of  the  Learning  Objective  (05.10) 

Those  listed  in  the  Task  1 Description  of  the  Microfiche  Technology 
will  be  sustained  and  repeated  here.  In  general,  they  are: 

i 

Reference  knowledge. 

Enabling  knowledge. 

Some  task  formats  to  limited  stages  of  learning. 

Some  transactional  procedures  to  limited  stages  of  learning. 

Further  specifications  of  objectives  will  be  made  in  the  respective 
topics  cited  above  in  later  sections  of  this  analysis. 

Preliminary  Evaluation  of  Importance  of  Benefits  (05.11) 

Further  examination  of  the  potential  benefits  of  the  technology  as 
applied  to  shipboard  training  has  sustained  the  evaluations  of  im- 
portance made  earlier. 

1.  The  training  objectives,  if  feasible  in  this  environment, 
are  at  least  of  moderate  importance  and  could  become,  if 
sufficiently  widespread,  of  very  substantial  importance. 

2.  The  magnitude  of  the  benefit,  if  microfiche  became  a key 
factor  in  the  practicality  of  widespread  shipboard  training 
in  job  knowledges  and  formats,  could  assume  very  substantial 
importance  to  the  Navy.  This  is  on  the  assumption  that  some 
x percentage  (to  be  estimated)  of  all  shore-based  training 
could  be  shifted  to  shipboard  training. 

Application  of  Microfiche  to  Skill  or  Knowledge  Training  (05.12) 

Assessment  will  be  made  both  in  the  knowledge  branch  and  the  skills 
branch  of  the  analytic  procedure. 

Task  Structure  Elements  in  Skills  Relevant  to  M/F  Instruction  (05.13) 

The  assumption  is  that  the  M/F  device  or  viewer  is  advanced  manually 
frame  by  frame.  Advance  may  be  relatively  random  with  respect  to 
frames  on  a given  M/F  card.  The  content  of  the  frames  on  a card  may 
be  designed  so  that  the  center  row  across  the  card  contains  the  primary 
level  of  instruction  sequence.  If  the  student  requires  more  procedural 
information  at  a given  frame  in  the  primary  row,  he  moves  to  the  frame 
a row  above  or  below  the  primary  frame.  Each  row  contains  a standard 
category  of  information  to  supplement  the  primary  row.  A given  row 
may  contain  workbook  information,  or  test  questions  for  the  primary 
frame  in  its  column.  Depending  on  his  needs,  the  student  can  obtain 
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supplemental  instruction  at  any  point,  and  test  himself.  (These  pro- 
cedures have  been  described  in  an  R&D  report,  and  demonstrated  to  be 
effective.) 

The  question  in  this  topic  is  what  skills  or  components  of  skills  can 
be  learned  merely  from  static  displays,  without  timing  constraints, 
and  where  the  student  expresses  his  response  output  symbolically  and 
must  himself  determine,  from  displayed  information,  whether  his  re- 
sponse was  adequate  or  inadequate.  (Separate  analysis  will  be  given 
to  knowledge  and  task  format  objectives.) 

Goal-Image.  These  can  be  learned  to  the  extent  that  the  goal-image 
can  oe  displayed  realistically  in  the  form  of  text,  diagram,  or  pic- 
ture. The  image  represents  the  appearance  of  a state  of  affairs  when 
the  task  is  properly  performed  as  contrasted  with  outcomes  when  the 
task  is  improperly  performed.  The  student  learns  to  make  the  discri- 
minations. In  the  process  of  instruction,  he  may  be  given  analytic 
guidance  for  making  these  judgments,  but  ultimately  he  may  have  to 
recognize  all-at-once  whether  the  picture  is  correct  or  incorrect, 
and  specify  what  should  have  been  done,  or  what  should  now  be  done 
to  correct  the  goal -picture. 

Some  examples  of  goal -images  learnable  by  M/F  are: 

Correctly  filled-in  questionnaires  of  a given  kind,  or  other 

transactional  documents;  dietary  menus. 

Sighting  pictures. 

Wave  shapes  shown  on  test  instruments. 

Diagrammatic  and  verbal  descriptions  of  battle  plans. 

Except  for  the  first  item,  all  of  these  are  examples  of  learning  that 
call  for  Identifications  and,  in  some  cases,  Interpretations.  The 
first  example  in  the  list,  although  legitimate  to  Goal-Image,  may  be 
equally  legitimate  as  an  instructional  factor  in  "decision-making"  and 
"constructing"  formats.  The  fact  that  examples  for  Goal-Image  teachable 
by  M/F  can  dissipate  into  other  more  inclusive  categories  of  learning 
operations  suggests  that  we  abandon  this  as  a search  descriptor,  and 
expect  little  loss  in  the  range-of-effect  picture. 

Scan-Detect.  Only  the  scanning-detecting  in  static  images  will  be 
applicable  and  this  element  should,  therefore,  probably  be  included 
as  members  of  a set  extending  to  Identify.  The  types  of  task  content 
for  this  category  applicable  to  M/F  clearly  are  maps,  charts,  blue- 
prints, semi -symbolic  representations  such  as  radar  displays,  and 
materials  used  in  photo-reconnaissance  interpretation.  Knowing  the 
applicability  of  these  examples  of  content  should  help  the  investi- 
gator judge  whether  to  accept  or  reject  the  hits  made  by  the  Scan- 
Detect-Identify  search  on  job  tasks  or  on  training  course  descriptions. 
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The  large  number  of  training  and  test  examples  that  can  be  put  into 
small  physical  space  on  M/F  for  individual  use  make  this  an  excell- 
ent medium.  This  value  is  augmented  by  the  student's  ready  access 
to  supplementary  guidance  for  each  problem  presented.  Useful  skill 
levels  in  scan-detect-identify  are  acquired  only  when  there  is  the 
speed  and  reliability  of  performance  that  can  be  attained  only  by 
many  hundreds  of  practice  examples.  This  is  not  a trivial  training 
function  where  it  is  applicable. 

Identify.  Part  of  this  task  function's  applicability  has  been 
treated  in  the  Scan-Detect  topic.  What  is  additionally  applicable  to 
the  learning  of  task  information  consists  of: 

Nomenclature:  naming  and  identifying  work  objects,  processes 

and  characteristics;  specifying  or  recognizing  their  location 
and  physical  relationships  to  other  work  objects. 

The  learning  of  task  nomenclature  is  often  not  a trivial  operation  and 
can  be  a useful  substrate  to  learning  the  procedures  that  apply  to  the 
work  objects  denoted  by  the  nomenclature.  M/F  and  a manually  operated 
device  may  not  be  ideal  for  learning  this  kind  of  content,  but  it  can 
serve  adequately,  especially  if  supported  with  effective  student  test- 
ing operations. 

Interpret.  Unless  the  task  is  time-driven,  the  capacity  for  M/F  to 
deliver  hierarchically  organized  and  accessible  information  should 
make  it  useful  for  training  up  to  at  least  intermediate  competence 
in  the  "interpreting"  demands  of  many  tasks.  A situation  may  be 
displayed,  and  the  student  is  required  to  interpret  its  significance 
either  as  to  the  circumstances  that  caused  the  pattern  of  conditions 
to  appear,  or  in  the  judgmental  aspects  of  selecting  a useful  task 
response.  It  is  perhaps  more  useful  for  the  student  to  be  exposed  to 
a large  number  and  wide  variety  of  situations  requiring  interpreta- 
tions, than  to  engage  him  in  profound  analysis  of  a limited  number 
of  examples.  A substantial  aspect  of  his  training  and  performance 
is  in  his  ability  to  hold  in  mind  the  context  of  information  upon 
which  the  interpretation  is  based. 

Examples  that  would  apply  to  M/F  instruction  are: 

A pattern  of  symptoms  in  a device,  where  its  interpretation  could 
result  in  determining  (a)  the  cause  of  the  symptoms  and/or  (b) 
what  capabilities  the  device  still  retained  for  operational  per- 
formance. 

A pattern  of  enemy  actions  is  described  from  which  the  enemy's 
Intent  and  objective  is  to  be  inferred. 

A pattern  of  behavior  data  from  an  incumbent  from  which  his 
capabilities  and  motivations  are  to  be  inferred. 
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A pattern  of  signals  is  apprehended  through  a noisy  channel 
from  which  the  entire  message  Is  to  be  reconstructed. 

A pattern  of  symbols  Is  displayed  on  a map,  or  other  type  of 
diagram  is  presented  from  which  the  actual  physical  charac- 
teristics of  the  objects  symbolized  are  to  be  interpreted  so 
that  they  would  be  recognizable  if  present. 

The  foregoing  would  be  guides  for  identifying  training  objectives 
and  their  contexts  that  might  be  identified  under  "Interpretation" 
but  require  additional  judgmental  criteria  for  selection  as  candi- 
dates for  training  with  M/F,  at  least  to  intermediate  stages  of 
competence  in  the  task  element. 

Interpret.  Where  the  meaning  or  significance  of  a pattern  of  cues 
must  be  translated  into  an  inference  about  a probable  state  of  affairs, 
and  there  is  little  interaction  allowed  in  the  task  itself  for  speci- 
fic Inquiry  for  further  data,  M/F  can  provide  useful  instruction.  But 
In  the  contexts  of  relevance  to  M/F  capabilities,  the  content  of  such 
instruction  can  more  usefully  be  characterized  as  "task-enabling 
knowledge." 

What  will  be  excluded  from  consideration  for  M/F  training  will  be 
such  meanings  of  interpret  as:  interpret  from  one  language  to 

another,  and  Morse  code  Interpreting.  Although  it  is  possible  to  use 
a static  medium  for  learning  many  aspects  of  a language  translating 
skill,  and  perhaps  especially  If  It  is  coupled  to^an  audio  input 
function  to  the  student,  the  M/F  medium  is  apt  to  be  highly  ineffi- 
cient. (If  there  are  empirical  demonstrations  to  the  contrary,  the 
data  should  have  primacy  in  guidance.) 

The  interpreting  function  draws  upon  task-enabling  knowledge,  so  we 
can  expect  overlap  of  Instructional  content  that  is  drawn  from  a 
task  requirement  using  a term  like  "Interpret"  or  deduce  or  figure 
out  or  similar  expression  with  content  that  Is  drawn  from  descrip- 
tions of  job  and  task  knowledge,  especially  where  the  criterion  of 
"enabling"  has  been  used  as  a filter  of  content. 

Note:  The  brevity  of  treatment  of  a topic,  such  as  the 

present  one,  should  not  be  misinterpreted  to  mean  that 
little  or  unimportant  instructional  material  will  be 
derived  from  it.  In  this  case,  it  is  the  contrary.  In 
a fair  number  of  tasks  there  can  be  substantial  instruc- 
tional content  derived  that  is  applicable  to  "interpret." 

Procedures  Performance.  Microfiche  technology  --  with  the  simple 
viewing  mechanism  postulated  for  this  assessment  --  can  train  in  any 
procedures  which  have  been  amenable  to  conventional  Programmed 
Instruction.  Thus,  any  PI  on  a paper  medium  could  be  readily  trans- 
lated into  the  M/F  medium  with  somewhat  greater  flexibility  in  sequen- 
cing and  material  than  is  practical  (in  terms  of  the  student)  with 
paper. 
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Note  that  "formats"  for  procedures  will  be  treated  in  a later  section. 
In  this  section,  the  concern  is  about  instruction  in  skill  learning. 

M/F  lends  Itself  to  instruction  in  procedures  that  are  restricted  to 
symbolic  materials  and  symbolic  operations.  It  would  be  somewhat 
less  efficient  than,  say,  a computerized  capability  of  answer  analysis 
and  branching  to  specifically  suitable  remediation.  (One  can  be 
overly  preoccupied  with  this  kind  of  efficiency  and  neglect  comparing 
the  real  benefits  with  the  very  real  costs!)  Examples  of  symbolic 
materials  and  operations  are: 

Learning  and  applying  calculational  rules. 

Navigational  calculations  with  various  kinds  of  input  data 
as  given. 

Performing  statistical  analyses  and  interpretations. 

Filing  procedures;  setting  up  files. 

Most  clerical  transactional  procedures  (see  the  analytic 
definition  of  "transaction"  in  the  Appendix  A section 
called  Task  Formats  as  a guide  for  applicable  task 
s tructures  here.) 

Decide.  At  least  preliminary  training  (at  the  decision  format  level) 
can  be  given  in  classes  of  decision  that  are  well -structured  (all 
variables  relevant  to  the  decision  class  readily  prepresented), 
where  the  apprehending  of  the  problem  Information  is  not  dependent 
on  absolute  time  values,  and  where  the  decision  itself  is  not  time 
dependent.  Furthermore,  because  of  the  limitations  of  the  medium 
and  device,  the  problem  information  must  be  presented  in  a form 
where  the  variables  have  already  been  symbolized  (the  learner  does 
not  have  to  Identify  the  variable  and  the  value  in  a complex 
perceptual  environment)  and  usually  the  learner  has  only  to  select 
from  a restricted  set  of  response  alternatives. 

Static  display  media,  while  able  to  give  limited  practice  to  the 
judgmental  aspects  of  linking  a problem  situation  to  a decision 
response,  may  give  extensive  "task-reference  knowledge"  by  giving  the 
learner  the  Implications  of  selecting  various  choices  in  a given  kind 
of  situation.  Acquiring  Information  that  enables  judgment  in 
decision  making  is  often  difficult  in  real  life  because  the  Incumbent 
frequently  does  not  see  the  outcome  of  his  choice,  or  if  he  does,  so 
much  time  separates  his  making  the  decision  from  learning  the  outcome, 
that  learning  is  inefficient  and  unreliable.  Thus,  although  the 
learner  may,  with  this  medium,  acquire  only  limited  levels  of  skill 
in  actual  decision  making,  he  can  acquire  verbal  and  conceptual 
information  that  will  assist  in  that  skill.  In  summary  of  this 
point,  he  can  be  given  practice  enabling  him  to  anticipate  outcome 
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probabilities  from  selecting  a given  choice,  as  well  as  in  the 
learning  of  the  variety  of  options  he  may  have  available  from  which 
to  choose.  These  might  be  called  "task-enabling  knowledges"  for 
decision  making. 

Some  decision-making  task  formats  may  be  learned  with  the  M/F  medulm 
and  the  simple  display  device. 

A qualification  should  be  made  here.  The  development  of  Instruction- 
al content  that  will  generate  useful  practice  and  learning  even  In 
the  preliminary  aspects  of  decision  making  (coping  with  uncertainty 
In  problem  situations  varying  In  their  degree  of  explicit  structure), 
requires  a high  level  of  teaching  art  and  insight  Into  human  decision 
processes.  This  level  of  talent  is  likely  to  be  rare.  A practical 
decision  In  assessment  might  well  be  to  forego  "decision  making"  as 
a task  element  for  which  M/F  and  the  simple  display  device  Is  a 
fruitful  application. 

Construct.  The  same  arguments  developed  for  Decide  apply  here, 
except  that  the  limitations  of  the  medium  and  the  dependence  on  the 
art  of  the  Instructional  content  designer  Is  even  greater. 

"Construct"  Is  not  a promising  candidate  except  for  perhaps  very 
rare  tasks.  The  learning  of  task- reference  knowledge  to  support 
skills  In  a given  kind  of  constructing  can  probably  be  done  with 
M/F,  although  not  with  ideal  efficiency. 

Microfiche  and  simple  viewers  hold  little  or  no  promise  for  instruc- 
tion In  the  following  task  elements:  Tracking,  Motor  Performance, 
Interpersonal  Interaction,  Recall  of  Task  Cycle  Information,  or 
Recall  of  Enabling  Information.  The  last  two  task  elements  are 
processes  subsumed  under  other  rubrics:  M/F  does  not  offer  as  a 

medium  any  special  properties  for  either  type  of  recall  that  are 
not  shared  or  utilized  by  competing  media. 

In  summary,  we  have  the  following  descriptors  that,  with  some 
qualifications  In  all  cases,  can  be  used  to  search  the  data  base 
description  of  courses  that  Include  skill  (or  "practical  factors") 
training. 

Goal -Image:  heavily  restricted  to  tasks  dealing  with 

verbal  formats  and  content,  some  perceptual  skills  In 
differentiation,  and  diagrammatic  representations  of 
task  content. 

Scan-Detect-Identlfy:  cognitive  content  In  maps,  charts, 

blueprints,  semi-symbolic  representations  such  as  radar 
displays,  and  photo-reconnaissance. 

Identify:  nomenclature,  identifications  and  locations  of 

work  objects.  (This  Item  will  be  repeated  In  "Enabling 
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Knowledge".)  Also  the  identification  of  patterns  of 
elements  that  in  combination  become  an  identifiable 
entity. 

Interpret:  enabling  knowledges  that  enable  the  operator 

to  determine  the  meanings  implied  by  patterns  of  task 
cues.  (Note  a distinction  between  identifying  a pattern 
and  Interpreting  what  the  pattern  means). 

Procedure  Performance:  in  skill  learning,  restricted  to 

early  and  preliminary  levels  of  symbolic  tasks  and  materials: 
calculations,  navigational  calculations  and  logic;  clerical 
transactions  — as  examples. 

Decide:  at  least  preliminary  skill  training  beyond  the 

level  of  decision  formats,  but  in  tasks  where  the  actual 
problem  information  is  presented  symbolically  as  cogni- 
tive content  and  not  time-dependent  such  as  the  decision 
to  abort  a landing  on  a carrier,  or  where  the  context  of 
problem  and  resource  is  so  complex  that  it  cannot  usefully 
be  presented  in  static  displays.  Best  bets  are  clerical 
transactions  which  demand  "judgment"  in  the  face  of  some 
Indeterminacy  and  uncertainty. 

Construct:  only  task-enabling  formats  for  constructing 

a class  of  entity. 

No  additional  task  elements  seem  appropriate  to  the  M/F  medium 
except  as  they  have  been  already  treated  in  other  task  elements 
or  in  the  enabling  knowledge. 

Identify  Relevant  Jobs  and  Tasks  (05.14) 

In  the  present  assessment,  this  step  will  be  skipped.  The  reason  is 
that  the  major  targets  for  M/F  will  be  "knowledge"  courses,  and  the 
"knowledge"  component  in  practical  skills  courses. 

If,  however,  a job-task  data  base  were  to  be  learned,  the  task 
element  names  selected  in  05.13  would  be  applied  with  the  following 
restrictive  specification: 

Equipment  and  objects  used  in  the  task:  "symbolic". 

Reference  information  used  in  performing  the  task:  "symbolic; 

AND/OR  static  perceptual . ll 

Relevant  Stage  of  Learning  (05.15) 

Different  classes  -of  learning  objectives  applicable  for  microfiche  as 
a medium  --  and  the  simple  viewer  --  can  be  learned  to  somewhat 
different  stages  from  the  point  of  view  of  their  contribution  to  task 
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performance  level.  Refer  to  the  objectives  selected  in  Subtask  05.10. 

Reference  Knowledge:  although  these  can  be  learned  to  reliable 
levels  as  knowledge,  practice  in  the  context  of  the  task  itself  is 
required  for  their  effective  integration  into  performance.  Thus, 
with  regard  to  a task  performance  criterion,  reference  knowledge  will 
be  learned  "to  the  verbal  level11. 

Enabling  Knowledge:  the  same  applies  here  as  for  Reference  Knowledge. 

Stage  of  learning  will  be  to  the  verbal  level,  except  in  the  case  of 
Transactional  Procedures,  treated  below. 

Task  Formats:  with  respect  to  symbolic  or  perceptual  tasks,  the  M/F 
training  could  take  the  student  to  "Performing  Task  Components  with 
Guidance." 

Transactional  Procedures:  the  qualifications  made  in  05.13  should  be 

recalled  here.  The  tasks  that  fit  the  qualifications  can  probably  be 
learned  to  the  level  of  "Doing  the  Entire  Task/Job  Procedurally: 

Barely  Acceptable  Mastery?1  The  student's  reaching  this  level  will 
depend,  among  other  things,  on  the  degree  to  which  the  practice  con- 
tent has  adequately  sampled  the  range  of  problem  situations  in  the 
real  job  environment. 

All  Procedural  Operations:  although  the  initial  objectives  for  M/F 

usage  did  not  identify  refresher  training,  M/F  could  be  given  on 
jobs  currently  being  held  by  the  Incumbent-student  and  especially 
over  knowledges  and  procedures  that  may  be  rarely  used.  Refresher 
training  may  also  be  given  to  students  who  have  completed  courses  of 
training  but  who  have  not  been  assigned  to  jobs  In  which  they  can 
exercise  that  training.  The  present  assessment  will  not  include 
refresher  training  in  the  analysis  because  Its  Inclusion  could  cloud 
the  main  Issue  of  M/F  as  primary  training.  But  Its  shipboard  use  for 
refresher  training  could  be  a strategy  for  widening  Its  usage  to 
primary  training,  and  thus  gain  acceptance  progressively. 

Refresher  training  would  be  applicable  to  all  Incumbents  in  their 

Sresent  job  positions.  It  would  also  apply  to  all  incumbents  who 
ad  training  In  duties  Into  which  they  had  not  yet  been  assigned. 

Identify  Relevant  Courses  (05.16) 

Some  labor  saving  strategies  may  be  used  In  determining  courses  and 
course  content  applicable  to  M/F  Instruction.  Some  of  the  following 
repeats  advice  offered  In  the  Introduction  to  Task  5 

1.  Courses  that  are  primarily  knowledge  and  theory  background 
for  practical  skills  training.  Except  for  testing  and 
student  evaluation,  virtually  all  Instructional  functions 
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provided  by  lecture  and  paper  can  be  taken  over  by  M/F. 
Whether  the  motivational  functions  of  the  classroom 
environment  (such  as  they  are)  can  be  replaced  in  the 
self-teaching  environment  would  require  empirical  inquiry. 
The  equivalent  in  classroom  course  hours  (which  normally 
does  not  include  self-study  hours)  in  M/F  study  hours 
might  need  to  be  determined  empirically  in  any  benefits 
evaluation,  although  student  time  of  shipboard  given  to 
study  may  have  a different  cost  and  value  than  student 
time  in  shore-based  training. 

2.  Skills  courses  potentially  applicable  to  shipboard  training. 
Training  Units  Within  Courses  (05.17) 

Within  skill  courses,  the  following  training  units  or  objectives  will 
be  applicable  to  M/F  instruction. 

1.  Any  type  of  knowledge  as  identified  in  Subtask  05.19. 

2.  Procedural  and  transaction  formats  as  defined  in  Subtask 
05.23. 

3.  Nomenclature  and  identifications  specific  to  task  and 
environment. 

4.  Task  structure  elements  as  summarized  at  the  end  of  the 
writeup  for  Subtask  05.13  in  this  assessment  report. 

Procedural  Comment.  A short-cut  for  making  estimations  can  be  based 
on  sampling  procedure.  Subdivide  all  skills  courses  considered  eli- 
gible for  shipboard  training  into  groups  that  deal  with  (a)  opera- 
tional skills  with  equipment;  (b)  maintenance  jobs;  (c)  service  and 
miscellaneous  jobs.  Including  clerical;  and  (d)  command  jobs.  Within 
each  group,  choose  at  random  half  a dozen  course  descriptions  at  the 
most  detailed  level  available.  Make  an  analysis,  objective  by 
objective,  of  each  Item  of  training  in  the  course  description  that 
fits  the  applicability  to  M/F  according  to  the  search  criteria  estab- 
lished in  Task  5.  Make  an  estimate  of  the  ratio  of  the  training 
hours  of  M/F  material  to  the  total  course  hours.  Thus  for  one 
course  In  a group,  the  M/F  applicable  content  may  be  25%  of  the  total 
hours;  for  a second  course  it  may  be  22%;  for  a third  28%.  If  there 
is  relatively  little  practical  variation  among  the  samples  of  these 
ratios  for  the  group  of  course,  choose  the  representative  value  among 
the  sample  ratios  and  extrapolate  to  the  ratio  value  for  the  entire 
set  of  courses  in  that  type  of  course.  If  the  sample  values  fall 
within  a 10%  range  for  90%  of  the  sample  members.  It  would  seem  safe 
to  choose  a mean  or  median  of  the  sample  values  as  representative  of 
that  class  of  courses. 
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In  some  job  and  training  specifications,  the  Navy  has  already  made 
a division  between  knowledge  content  and  practical  skills  content. 
Thus,  In  the  Personnel  Qualification  Standards,  the  100  series  and 
the  200  series  are  directly  applicable  to  knowledge  training,  hence 
to  M/F. 

In  the  format  used  for  the  Navy's  "Occupational  Standards  Workshop 
Worksheets,"  the  knowledge  factors  are  separated  from  the  Practical 
Factors  (skills). 

Where  job  specifications  of  these  kinds  are  reflected  in  the  specifi- 
cations of  training  objectives,  the  identification  of  M/F  applicable 
training  is  tedious  rather  than  complicated. 

As  a course  applicable  both  to  shipboard  training  and  to  the  M/F 
medium  is  identified,  the  projected  numbers  of  students  for  that 
course  is  noted.  These  numbers  will  be  used  to  put  scale  Into  the 
magnitude  of  the  potential  M/F  application. 

Assessing  the  Benefit  Pattern  (05.18) 

The  two  Innovations  that  are  jointly  to  be  assessed  consist  of  the 
substitution  of  M/F  medium,  plus  simple  viewer,  for  paper  in 
instructional  materials  — with  the  unassessed  by-product  in  M/F 
substitution  for  many  Navy  technical  manuals  now  on  paper  — and  the 
substitution  of  shipboard  training  for  a large  proportion  of  shore- 
based  training.  In  both  of  these  factors,  the  major  benefit  to  be 
expected  Is  a savings  in  costs.  In  part  at  least,  the  ability  to 
reduce  the  bulk  of  training  material  from  that  of  paper  to  that  of 
M/F,  makes  it  practical  as  a "necessary"  condition  for  shipboard 
training.  The  range  of  training  (In  terms  of  courses  and  jobs 
affected)  and  the  Importance  of  what  can  be  learned  with  It  makes 
shipboard  training  feasible. 

But  shipboard  training  will  be  shown  to  be  practical  (assuming  that 
conflict  with  operational  exigencies  has  been  adequately  managed)  if 
cost  factors  for  shipboard  training  compare  favorably  with  costs  for 
shore-based  training.  Making  this  comparison  requires  Invoking  the 
cost  model  for  shore  training  and  an  equivalent  cost  model  applicable 
to  shipboard  training. 

A factor  not  previously  examined  about  shipboard  training  Is  that  of 
supplemental  devices,  including  samples  of  operational  equipment,  for 
training  In  actual  skills  (the  practical  factors)  for  which  knowledge 
and  format  learning  is  merely  a precursor.  Either  such  skills- 
training equipment  Is  on  the  ship,  or  the  student  must  acquire  the 
practical  skills  In  shore-based  schools,  or  the  "skills"  factor  will 
be  satisfied  (for  advancement  purposes)  by  verbal  learning.  Including 
descriptions  of  procedures. 
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The  most  realistic  assumption  is  that  only  those  skills  courses  will 
be  taught  on  a ship  for  which  there  are  copies  of  operational  equip- 
ment on  which  students  can  learn  skilled  performance.  But  this 
assumption  In  practice  has  costs,  possibly  Including  operational 
costs  and  risks,  which  complicate  the  shipboard  training  picture, 
beginning  with  acceptability  to  the  commander  of  the  ship. 

Let  us  assume  that  cost  comparisons  have  been  performed  and  qualified 
by  judgments  of  probability  of  achieving  the  cost-benefits  of  M/F. 
Let's  now  make  a pass  at  the  questions  in  05.18. 

1.  Reduction  in  training  time.  None  expected.  If  M/F 
Increases  training  time  as  compared  with  shore-based 
training,  the  costs  are  not  relevant  to  any  budget^ 

2.  Reductions  in  attrition  rate.  There  are  somewhat  different 
factors  that,  a priori,  seem  to  work  for  and  against 

. attrition  rates  in  shore-based  conventional  training 
and  shipboard  training.  Shipboard,  the  trainee  works  at 
his  own  rate,  and  there  is  little  or  no  penalty  if  he 
takes  twice  as  many  study  hours  to  complete  an  objective 
to  criterion  mastery  as  some  other  student.  On  the  other 
hand,  he  may  have  less  direct  association  with  Instructors 
on  shipboard  who  could  help  him  past  a learning  obstacle, 
or  to  motivate  him  individually  to  keep  on  trying.  All 
together,  one  set  of  factors  seems  to  cancel  out  the  other. 

3.  Reductions  in  student  aptitude.  Because  there  is  little 
expense  (as  compared  with  shore-based  training)  In 
conmittlng  a student  to  shipboard  training.  It  Is  feasible 
to  lower  the  aptitude  levels  for  shipboard  training  and 
take  a greater  chance  on  the  student's  success  in  com- 
pleting the  course  on  the  assumptions  that  (a)  aptitude 
tests  have  substantial  standard  errors  of  prediction  and 
(b)  the  student  can  take  more  time  to  complete  the  course 
requirements.  This  policy  Ignores  the  possibility  that 
there  is  a relationship  between  an  aptitude  score  and  the 
ultimate  level  of  the  incumbent's  skill  no  matter  how  much 
training  he  was  given.  Any  estimate  that  student  aptitude 
could,  In  general,  be  somewhat  reduced  because  of  the  M/F 
shipboard  capability,  would  have  to  be  close  to  zero  bene- 
fit. 

There  is  this  proviso,  however.  It  has  been  said  that  in 
many  cases  the  difficulty  of  the  cognitive  (knowledge) 
materials  in  training  such  as,  for  example,  "Theory  of 
Operation,"  is  one  major  basis  for  many  student's  attrition. 
If  success  in  such  knowledges  has  a low  correlation  with 
eventual  success  on  the  job,  then  any  training  procedures 
which  help  the  student  get  past  his  formal  knowledge 
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requirements  will  Increase  his  opportunities  for  getting 
Into  practical  training  and,  thus,  to  the  job.  If  this 
argument  Is  justified,  then  the  flexibility  offered  by 
Individualized  paths  through  M/F  would  lower  the  aptitude 
requirement,  perhaps  by  substantial  amounts. 

4.  Supplemental  benefits.  Shipboard  training  may  have  highly 
useful  side-benefits  to  the  ship  as  well  as  to  the  man. 

There  Is  at  least  Indirect  psychological  justification  for 
the  belief  that  "learning  activities"  and  "job  activities" 
should  not  be  separate  blocks  of  activity,  but  that  the 
learning  process  and  the  motivation  to  learn  should  be 
continuous  In  a person's  lifetime.  The  intellectual 
struggle  to  learn  something  new  during  prescribed  hours  of 
the  day  may  well  reflect  In  greater  alertness  and  inquiry 
In  the  hours  spent  on  the  job,  even  though  the  content  of 
what  Is  being  learned  differs  widely  from  the  content  of 
the  job.  Opinions  of  this  kind  remain  to  be  empirically 
validated  across  wider  populations  than  merely  the  pro- 
fessionals where.  In  industry,  this  assumption  Is  being 
applied.  People  take  courses  as  a part  of  their  work  day, 
and  presumably  some  of  the  stimulation  rubs  off  into  their 
work. 

Like  many  really  important  values  and  "benefits",  this 
"stimulus  to  learn"  Is  not  susceptible  to  quantitative 
assessment.  In  degree,  it  would  be  applicable  to  all  job 
Incumbents  who  were  also  students,  and  apply  to  all  jobs 
and  tasks.  It  would  have  at  least  a moderate  importance  to 
the  quality  of  performance  of  the  Navy  mission,  the  morale 
of  the  crew,  and  presumably  of  the  reenlistment  rate  of 
desirable  personnel. 

This  factor  should  probably  be  stated  as  a qualitative  plus 
to  the  decision  maker  rather  than  In  a quantitative  format. 

Types  of  knowledge  (05.19,  05.20  and  05.21) 

Microfiche  as  a medium  and  the  simple  viewer  as  the  device,  supple- 
mented by  periodic  performance  testing  with  other  means,  can  be 
applicable  to  the  learning  of  any  type  of  knowledge  for  which  the 
student  Is  motivated  to  learn.  For  some  types  of  knowledge  learning, 
there  may  be  other’ Interactive  arrangements  that  can  be  more  efficient. 
But  the  skill  with  which  the  Instructional  content  Is  designed,  dis- 
played,  and  sequenced  may  be  more  Important  than  any  combination  of 
other  factors  for  efficiency  In  learning  and  effectiveness  in  reten- 
tion and  application. 

All  classes  of  Reference  Knowledge  (05.20)  and  Enabling  Knowledge 
(05.21)  are  applicable.  The  more  significant  classes  of  knowledge 
under  these  rubrics  are: 
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EXHIBIT:  ETAM  EVALUATION  OF  MICROIMAGE  TECHNOLOGY  (Continued)... 


System  purposes. 

Contexts  of  operation. 

Theory  of  operation. 

Enabling  facts  and  prescriptive  rules. 

Concepts  and  principles. 

Nomenclature,  identification  and  locations  of  work  objects. 
Procedure  descriptions. 

Types  Of  Task  Formats  (05.23) 

The  following  types  of  format  are  susceptible  to  learning  from  the  M/F 
medium  and  the  simple  viewer.  This  combination  is  somewhat  more  flexi- 
ble for  branching  purposes  than  is  paper,  but  far  less  so  than  a com- 
puter-driven display  terminal.  The  branching  requirement  is  essential 
for  enabling  the  student  to  try  out  various  alternatives,  and  to  deter- 
mine the  Implications  of  various  alternatives,  as  well  as  to  present 
to  him  contingencies  that  may  be  associated  with  a "standard"  situation. 

Procedural  Formats:  the  general  sequence  of  actions  whereby 

something  gets  done. 

Transaction  Formats:  preskill  and  skill  learning  in  performing 

more  or  less  routinized,  well -structured  transactions  (often  of 
a clerical  type)  that  involve  filling  out  of  forms,  or  trans- 
lating Information  from  input  forms  into  output  forms  with  some 
processing  judgment. 

Decision  Formats:  learning  the  variables  of  problem  and  re- 

source  to  keep  in  mind  and  optimize  in  choice-making  activities. 
Extent  to  which  the  student's  management  of  "branching"  opera- 
tions in  M/F  enables  skill  development  in  use  of  the  format  on 
task  samples  will  probably  depend  on  the  specifics  of  the  deci- 
sion-making task  and  information  content.  Decisions  of  the 
clerical  type  associated  with  transaction  formats,  identified 
above,  can  almost  certainly  be  done  with  M/F  because  the  task 
usually  presents  limited  alternatives. 

Conclusion  to  the  Assessment  Exercise 


In  many  ways,  the  problem  of  M/F  and  shipboard  training  is  an  example  that  is 
not  representative  of  the  primary  objective  of  ETAM  which  is  rather  consistently 
pointed  towards  practical  skill  training.  But  perhaps  this  is  why  it  may  be  a 
useful  test  of  the  procedure. 
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An  attempt  has  been  made  to  express  not  merely  assessment  outcomes  — which 
could  be  done  in  a very  few  pages  — but  the  lines  of  thinking,  with  the  format 
of  ETAM  as  a guide,  to  the  conclusions  made.  It  should  also  be  noted  that  the 
conclusions  reached  had  the  benefit  of  almost  no  empirical  data  given  in  the 
Task  1 statement,  as  to  the  capabilities  of  M/F,  other  than  that  it  could  serve 
as  a medium  at  least  as  good  as  paper  (and  other  static  presentation  media)  for 
whatever  could  be  taught  and  learned  with  paper. 

One  intent  for  ETAM  was  that  assessors  would  be  led  to  think  constructively 
about  many  things  which  might  otherwise  be  ignored.  Even  though  the  ETAM 
formats  and  rationales  were  created  over  a number  of  months  and  with  the  help 
of  a number  of  problem  scenarios,  it  was  found  that  the  present  format  was  a 
significant  probe  and  stimulus.  It  is  hoped  that  the  reader  of  the  exercise 
will  vicariously  share  this  observation. 

An  attempt  has  also  been  made  to  show  examples  of  by-passing  procedural  steps 
when  common  sense  or  invention  justified  the  by-passing.  Although  the  ETAM 
procedures  emphasize  that  the  assessors  make  analytic  short-cuts  when  their 
objectives  are  clear  and  limited,  examples  may  be  more  forceful  in  communicating 
this  procedural  flexibility.  Analysis  should  not,  in  principle,  be  more  exten- 
sive than  Is  required  to  make  the  decision  which  is  the  objective  of  that  analy- 
sis. It  Is  possible,  of  course,  for  the  analysis  to  reveal  what  the  nature  of 
the  decision  should  be. 

Another  major  value  in  this  form  of  assessment  is  that,  by  being  structured 
and  Its  response  made  in  writing,  it  can  be  reviewed  by  others  who  may  be 
better  Informed  or  who  have  additional  insights.  These  may  contribute  to 
effective  modifications  leading  to  better  decisions  about  acceptance,  rejec- 
tions, or  "more  study."  But  perhaps  just  as  important,  the  added  qualifica- 
tions and  Insights  may  lead  to  the  superior  use  of  an  "innovation,"  or  better 
safeguards  against  its  misuse.  Thus,  the  assessment  process  itself  makes  con- 
tributions to  the  technology  of  education  and  training  under  conditions  favor- 
able to  the  direct  application  of  that  technology:  the  actual  practices  of 

instruction  and  of  learning. 
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SECTION  VI 

ETAM  VALIDITY  AND  UTILITY 


The  validation  of  a procedure  can  be  split  into  several  parts.  One  is 
the  extent  of  flexibility  and  adaptability  of  the  procedure  to  a variety 
of  conditions  and  objectives.  A second  is  whether  the  procedures  can  be 
followed,  and  with  what  difficulties,  uncertainties  and  pitfalls.  The 
third  aspect  of  validation  is  the  evaluation  of  how  good  the  results  of 
following  the  procedures  can  be. 

FLEXIBILITY 

It  is  unnecessary  to  crank  an  assessment  through  the  entire  ETAM  procedures 
in  order  to  present  data  to  the  decision  process.  The  description  of  the 
innovation,  as  structured  in  Task  1,  may  be  passed  directly  to  Tasks  3 and 
4 where  they  can  be  formatted  for  presentation  to  the  decision  maker.  Wher- 
ever the  pattern  of  the  benefits  projected  from  a proposed  innovation  suf- 
ficiently outweighs  the  costs  and  liabilities,  the  combined  patterns  may  by- 
pass the  remainder  of  the  data  collecting  and  analysis  steps,  and  be  pre- 
sented to  the  decision  maker. 

Task  5 assists  the  assessor  to  probe  into  the  fullest  potential  range  of 
effect  of  the  innovation  in  terms  of  extent  and  boundary  limits.  But  if 
at  any  stage  of  inquiry,  a sufficient  range  of  effect  has  been  determined 
to  justify  a benefits- to-cost  decision,  further  analysis  can  be  halted  and 
the  case  can  be  presented  to  the  decision  maker.  If  gross,  overall  estimates 
of  a benefit  are  justified,  (say  for  all  of  a given  type  of  course)  it  is 
unnecessary  to  undertake  analytic  work  for  each  individual  course  and  sum 
the  totals.  If,  however,  a decision  is  made  to  accept  the  innovation  and 
at  the  same  time  specify  a program  for  implementing  the  decision  in  prac- 
tice, the  full-scale  analysis  in  Task  5 should  be  made  for  that  purpose. 

Task  3 in  ETAM  poses  a number  of  key  questions  to  be  addressed  to  the  pro- 
posed innovation.  Even  one  crucial  negative  answer  can  justify  terminating 
further  assessment  because  the  innovation  has  failed  to  meet  a necessary 
condition  to  acceptance:  so  why  continue  further? 

The  assessor  is  invited  in  Task  8 to  anticipate  the  kind  of  data  presenta- 
tion and  analysis  that  the  decision  makers  will  want.  Some  will  prefer 
emphasis  on  the  financial  analysis  developed  in  Task  7,  others  will  con- 
sider the  output  of  the  benefit-cost  analysis  in  Task  6 as  the  more  impor- 
tant input  to  the  decision  process.  Furthermore,  the  decision  maker  may 
prefer  to  ignore  the  outputs  of  the  analytic  models  and  prefer  to  base  his 
decision  on  the  picture  of  the  raw  benefits,  costs,  liabilities  and  proba- 
bility data  developed  in  Tasks  1,  3 and  5;  as  well  as  on  several  alterna- 
tives which  may  have  been  developed  in  Task  2. 

This  flexibility  in  ETAM  procedures,  while  increasing  its  versatility  and 
responsiveness  to  the  needs  of  the  occasion,  also  increases  the  difficulty 
in  "evaluating"  those  procedures  according  to  fixed  criteria  on  quality  of 
outcome.  By  definition,  a rigid  entity  is  more  easily  evaluated  than  a 
flexible  entity. 
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It  is  possible  that  in  the  future  some  policy  makers  will  deplore  the  ex- 
isting options  for  bypassing  large  segments  of  formal  analysis  in  ETAM. 

The  existing  exits  can  be  blocked  off  readily  enough--but  the  choice  should 
lie  with  the  using  agency. 

VALIDATION  OF  THE  LOGICAL  STRUCTURE  OF  ETAM 

Formal  presentations  of  the  procedural  structure  of  ETAM  Tasks  1 through  8 
have  been  given  to  representatives  of  the  sponsor  and  to  associated  Navy 
agencies.  Although  constructive  comments  were  made  that  led  to  changes  in 
procedure  and  nomenclature,  no  fundamental  flaws  were  revealed  in  the  course 
of  these  reviews.  This  is  supportive  rather  than  conclusive  evidence  that 
flaws  do  not  exist,  but  they  do  not  seem  evident  to  inspection  by  experts 
in  training  systems  and  in  decision  systems. 

SCENARIO  APPLICATIONS  OF  THE  PROCEDURE 

ETAM  procedures  were  conceived  and  formalized  around  a number  of  hypotheti- 
cal "scenarios."  Several  of  these  exist  in  preliminary  working  papers,  and 
were  one  major  basis  for  conmuni cations  with  the  project  consultant,  Dr. 
Gagne. 

Section  V of  this  report  contains  a full-dress  evaluation  scenario  of  a 
hypothetical  Innovation  that  was  carried  through  all  tasks  up  to  and  in- 
cluding the  presentation  of  decision-making  content.  This  example  is  the 
most  comprehensive  "validation"  of  the  ETAM  procedure  to  this  date.  Each 
step  could  be  readily  performed.  Where  the  assessor  was  uncertain  about 
expected  outcomes,  this  uncertainty,  and  its  approximate  level,  could  be 
expressed  formally.  In  this  sense,  a major  objective  of  ETAM  to  get  as 
much  information  as  possible  that  was  available  to  the  assessing  team,  in- 
cluding the  expression  of  tentativity,  was  clearly  realized.  One  does  not 
need  to  get  stuck  and  be  unable  to  proceed  in  the  procedures  because  one 
lacks  absolute  data  at  any  given  point.  Ignorance  and  undertainty,  by  being 
expressed,  become  legitimate  evaluation  data. 

A second  scenario  treats  only  a portion  of  the  ETAM  procedures,  focusing 
more  on  the  selection  of  relevant  variables  and  on  the  effects  of  decision- 
making upon  the  management  and  measurements  systems.  It  was  not  so  much  a 
validation  of  the  usability  of  the  procedures  as  much  as  a vehicle  for  de- 
veloping an  awareness  by  decision  makers  of  the  potential  Implications  of 
their  decisions. 

A third  major  scenario  run-through  was  made  on  a problem  assigned  by  a 
representative  of  TAEG:  shipboard  training  using  micro-image  in  the  form 

of  microfiche  and  standard  viewer.  The  primary  objective  in  this  exercise 
was  to  test  the  usability  of  procedural  Tasks  1,  3 and  especially  5.  The 
results  of  that  exercise  are  also  reported  in  Section  V of  this  report. 

The  scenario  may  be  useful  in  showing  the  adapatability  of  the  framework 
to  short  cuts,  key  issues,  and  especially  to  the  classificatory  structure 
for  determining  limits  to  the  range  of  applicability  of  the  stipulated  pro- 
posal. If  functions  are  added  to  the  stipulated  innovation,  the  benefits 
conferred  by  these  functions  in  terms  of  extended  range  of  training  effect 
can  be  readily  identified. 


VI-2 


TAEG  REPORT  NO.  12-3 


A cost  and  business  assessment  was  not  required  of  the  microfiche  assess- 
ment exercise,  although  the  data  leading  to  that  phase  of  assessment  were 
pointed  to. 

Since  in  all  three  of  these  scenarious  the  assessors  were  by  necessity  also 
the  developers  of  the  procedures,  their  representativeness  of  future  ETAM 
users  may  be  questioned.  But  no  claim  is  made  that  assessment  based  on 
limited  information  applied  to  a wide  range  of  semi-knowns  will  be  simple 
or  automatic.  The  written  assessments  should  be  evidence,  however,  that 
the  procedure  can  be  made  to  work.  The  results  of  the  microfiche  assessment 
and  the  "Three-D  Procedure  Trainer"  assessment  revealed  that  the  structure 
of  the  procedure  fostered  both  creative  and  analytic  thought,  and  offered  a 
clear  pathway  through  a potential  jungle  of  possibilities. 

ETAM  should  be  validated  further  by  consultant  level  persons  and  decision 
makers  within  the  naval  training  function  all  of  whom  have  accepted  the 
stipulations  of  training  in  ETAM  concepts  and  procedures,  but  who  were  not 
part  of  the  development  team. 

This  should  lead  to  the  determination  of  the  levels  of  decision-making 
sophistication  that  are  practical  to  use  for  the  officials  responsible  to 
make  them.  But  because  ETAM  is  not  tied  exclusively  to  any  one  format  of 
presentation  to  the  decision  maker,  he  can  elect  to  use  what  is  congenial 
to  him. 

EVALUATIONS  OF  THE  PROJECT  CONSULTANT,  DR.  R.  M.  GAGNE 

Dr.  Gagne  is  an  international  authority  on  education  and  training  technolo- 
gies both  as  a researcher  and  as  an  application  development  consultant.  He 
has  shared  a continuing  interest  in  pragmatic  taxonomies  especially  in  edu- 
cational content  and  process.  He  has  been  Intimately  exposed  to  the  con- 
ceptual anatony  of  ETAM  during  all  stages  of  its  development.  His  counsel 
has  been  an  invaluable  contribution. 

He  concluded  his  detailed  and  exhaustive  review  of  the  content  of  the  Pre- 
liminary ETAM  report  with  the  following  overview  comments  in  his  letter  of 
June  11,  1975. 

On  Section  V (scenario  examples): 

"Seeing  the  procedure  followed  in  its  entirety  is  highly  Illuminating." 

On  the  procedures  and  report  overall: 

"My  thoughts  about  the  analysis  and  procedure  are  these: 

"1.  It  seems  to  me  that  the  system  proposed  is  definitely  a feasible 
one.  Innovations  and  other  proposed  changes  can  be  evaluated  in  this 
way. 
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"2.  The  procedure  described  is  highly  comprehensive  and  thorough. 

It  takes  Into  account  what  seems  to  be  the  full  range  of  factors  which 
might  influence  a decision,  and  which  might  contribute  to  such  a de- 
cision. It  encompasses  a very  great  range  of  materials,  hardware  and 
procedures  that  might  be  proposed  for  the  purpose  of  accomplishing  im- 
provements In  the  cost-effectiveness  ratio  applicable  to  training  opera- 
tions. It  considers  the  entire  range  of  Navy  jobs  or  job  functions  for 
which  training  might  be  given  or  proposed  to  be  given. 

"3.  As  a consequence  of  this  comprehensiveness,  it  is  a procedure  which 
might  be  described  as  elaborate.  I assume  that  a thorough  procedure  is 
what  the  Navy  wishes  to  have.  However,  it  needs  to  be  said  that  sim- 
pler procedures,  involving  fewer  steps,  can  readily  be  conceived.  What 
any  of  these  would  do  would  be  to  "cut  corners,"  that  is,  to  make  as- 
sumptions which  are  based  upon  informed  judgment,  rather  than  upon  de- 
tailed consideration  of  all  the  sources  of  "hard"  data  that  can  be  made 
available.  Of  course,  cutting  corners  also  means  approaching  more  closely 
the  procedures  that  already  currently  exist. 

"4.  Surely  one  of  the  most  valuable  contributions  of  the  proposed  sys- 
tem is  its  approach  to  the  task  of  estimating  range  of  effect.  It 
would  appear  that  there  has  been  no  truly  systematic  way  to  do  this  up 
to  now,  among  Navy  procedures.  The  Ideas  of  deriving  decisions  from 
(a)  the  categories  of  training  outcome,  (b)  the  levels  of  training, 
and  (c)  the  effects  on  job  performance  represent  techniques  of  great 
originality  and  also  of  practicality.  Existing  procedures  tend  to 
neglect  these  factors,  so  that  they  do  not  enter  into  the  estimates  of 
effects  of  innovations  in  systematic  ways.  As  a result,  inadequate 
estimations  of  the  effectiveness  portion  of  the  cost-effectiveness 
equation  are  likely  to  be  made.  The  inclusion  of  range  of  effect  esti- 
mation procedures  makes  possible  the  elimination  of  this  potential 
Inadequacy. 

"5.  The  additional  value  of  range-of-effect  procedures  should  also  be 
mentioned.  An  Introductory  statement  in  this  report  speaks  of  this 
procedure  as  follows  (p.  1-3):  "a  descriptive  and  analytic  terminology 

that  can  be  applied  to  data  base  descriptions  of  instructional  vehicles 
and  their  relevant  properties,  job  tasks  and  their  relevant  properties 
and  training  courses  and  content  within  training  courses."  The  broad 
applicability  of  such  a planning  tool  reaches  far  beyond  the  single 
activity  of  estimating  cost-effectiveness  of  an  innovation. 

"6.  The  procedure  seems  no  less  thorough  In  its  identification  and 
estimation  of  cost  factors.  Although  this  part  of  the  cost-effective- 
ness has  always  seemed  relatively  the  easier  (because  of  the  assumed 
existence  of  "hard"  data),  it  is  notable  that  the  sources  of  costs  in 
this  system  appear  to  be  fully  covered,  whether  they  seem  initially 
large  or  small.  This  aspect  of  the  system  also  contributes  in  no 
small  measure  to  its  precision. 

"7.  On  the  whole,  I consider  the  proposed  system,  to  which  I have 
contributed  only  as  a critic  and  evaluator,  a very  substantial  ac- 
complishment, of  outstanding  conceptual  soundness." 
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This  ends  the  quotations  from  Dr.  Gagne's  evaluations.  His  point 
number  3,  directed  at  the  preliminary  version  of  ETAM,  has  resulted 
in  changes  in  the  final  version  which  emphasize  and  underscore  oppor- 
tunities for  "short-cuts"  and  procedural  simplifications  where  the  data 
available  and  the  nature  of  the  decision  justify  short-cuts. 

Dr.  Gagne,  commenting  on  recommendations  in  the  report  for  further 
ETAM  effort,  endorsed  the  studies  proposed,  and  made  the  following 
assertion  relevant  to  evaluation  of  ETAM: 

"Developing  a computerized  assessment  model  ...  would  indeed  be  a 
challenging  task.  It  would  seem  that  ETAM  in  its  fully  conceived  form 
could  only  be  evaluated  in  this  way.  It  would  be  good  to  see  it  done." 


COMMENTS  ON  THE  DECISION  MODELS 

The  primary  validation  emphasis  was  on  the  use  of  the  decision  tree  frame- 
work outlined  in  Tasks  3,  4 and  6.  One  validation  question  was  concerned 
with  the  structure  of  the  decision  tree  itself.  A second  dealt  with  the 
approach  used  In  handling  utility  functions  with  multiple  attributes. 

As  a result  of  the  concern  that  the  original  decision  tree  did  not  permit 
variations  in  the  way  risk  reduction  projects  were  packaged  (and  thus  did 
not  permit  combinations  that  might  perhaps  be  more  cost-beneficial),  the 
decision  tree  was  redesigned  to  allow  the  decision  variable  to  be  calculated 
for  more  than  one  project  grouping.  Also,  the  new  tree  incorporated  a 
project  success  probability  not  available  in  the  original  tree.  The  new 
format  resulted  from  a review  of  the  decision  tree  framework  by  Dr.  William 
Giauque  of  the  Naval  Postgraduate  School,  Monterey,  California. 

A second  key  question  concerned  the  use  of  the  additive  (rather  than  the 
multiplicative)  model  for  multi -attributed  utilities.  Dr.  Giauque  was 
again  consulted.  A number  of  cautionary  notes  were  inserted  into  the  pro- 
cedures based  upon  his  recommendations,  however,  it  appeared  from  formu- 
lating several  analytic  possibilities  that  the  additive  form  would  be  valid 
for  the  high  majority  of  proposals  with  which  ETAM  would  be  concerned. 

It  is  recognized  that  the  scaling  methodology  for  the  "importance"  factor 
needs  further  study  and  refinement.  An  ETAM  predevelopment  study  has 
been  recommended  in  Section  I of  this  report. 

CONCLUDING  REMARKS 

The  logical  structure  of  ETAM  as  a conceptual  model  of  assessment  param- 
eters that  are  comprehensive  and  relevant  has  not  been  challenged.  These 
include  the  taxonomic  parameters  which  lead  to  (a)  identification  of  key 
rejection  factors  in  assessing  an  innovation  (Task  3);  (b)  potential  range 
of  effect  analysis  (Task  5);  (c)  presentation  of  assessment  data  to  the 
decision-making  process  (Tasks  4,  6,  7 and  8). 
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The  center  of  concern  has  been  the  implied  amount  of  labor  and  level  of  skill 
for  responding  to  the  apparently  large  number  of  steps  and  alternatives  (as 
offered  by  the  cost  and  benefit  taxonomies)  contained  in  the  procedures. 
Figure  VI-1  shows  rough  estimates  of  time  to  apply  ETAM  to  a range  of  innova- 
tions. The  bulk  of  the  voluminous  report  itself  is  intimidating.  The  ques- 
tion is  asked:  "Is  the  outcome  of  an  assessment  worth  all  this  trouble?" 

This  is  a cost-effectiveness  question,  and  can  be  answered  only  by  refer- 
ence to  the  costs  and  effectiveness  measures  of  existing  or  alternative 
practices  in  reaching  decisions  about  acceptance  or  rejection  of  innovations 
or  other  structural  changes  in  the  Navy's  education  and  training  picture.  A 
clear-cut  "decision  point"  often  cannot  be  ascertained  in  traditional  prac- 
tices but  rather  what  appears  is  a network  of  localized  pressure  points,  and 
unshared  rationales  and  objectives.  Neither  the  costs,  nor  the  benefits, 
liabilities,  and  risks  can  be  determined  from  such  processes. 

Revisions  in  the  ETAM  statements  of  procedure  in  the  preliminary  version 
have  emphasized  where  and  how  short-cuts  can  be  taken.  In  this  respect,  at 
least  opportunities  for  "simplification"  have  been  offered  and  justified. 

But  this  still  leaves  the  large  bulk  of  pages  making  up  the  procedural  "map" 
of  ETAM,  a formidable  tome  comparable  to  the  size  of  an  operating  and  mainte- 
nance manual  for  a piece  of  radar  gear.  It  is  proposed  that  ETAM  be  imple- 
mented on  a conversationally  interactive  display  terminal.  This  facility 
will  simplify  the  procedure  in  several  ways.  One,  it  suppresses  procedural 
information  that  is  not  pertinent  to  the  analytic  route— and  stage  in  that 
route— which  is  of  significance  to  the  assessor.  The  user  retrieves  selec- 
tively both  type  of  content  and  level  of  content.  Two,  it  relieves  the 
user  from  the  labors  of  computation.  Three,  not  only  are  appropriate  data 
formats  presented  for  the  entering  of  data  values,  but  these  data  values 
are  automatically  transcribed  to  output  formats  in  whatever  variety  have 
been  established.  The  computerized  implementation  would  enable  a "human 
factors"  validation,  as  contrasted  with  the  "logical  validation"  of  the 
ETAM  procedures. 

The  reduction  of  clerical  paper  shuffling  and  arithmetic  will  not,  however, 
eliminate  "complexity"  from  the  task  of  making  predictive  assessments.  This 
complexity  is  inherent  in  coping  with  uncertainty  and  incomplete  information 
in  the  problem  to  be  solved.  Finally,  human  judgments  of  "relative  impor- 
tance" and  "probability  of  given  outcomes"  will  have  to  be  made,  however 
simplified  the  format  structures  for  making  these  judgments.  In  general, 
the  more  important  and  far-reaching  the  outcome,  the  greater  the  range  of 
uncertainty  because  of  the  incompleteness  of  "hard  data."  It  would  be  dis- 
honest and  dysfunctional  to  disguise  real  dependencies  on  informed  human 
judgment  or  the  effort  and  cost  involved  by  the  exercise  of  that  judgment. 

The  act  of  creating  useful  information  where  It  does  not  exist  is  not  simple 
even  though  It  has  been  "simplified." 

Because  ETAM  was  explicitly  designed  as  a tool  to  serve  human  users,  its 
validation  cannot  be  independent  of  the  skills  and  diligence  of  the  user  in 
the  role  of  assessor  and  in  the  role  of  decision  maker. 
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Task  1 
Task  2 
Task  3 
Task  4 
Task  5 
Task  6 
Task  7 
Task  8 


TYPE  OF  INNOVATION 


SIMPLE 

AVERAGE 

COMPLEX 

1 Hr. 

4 Hrs. 

8 Hrs. 

(Not  Included) 

(Not  Included) 

(Not  Included) 

1 

2 

3 

16 

40 

160 

8 

24 

40 

12 

16 

24 

4 

8 

12 

8 

16 

40 

50  Hrs . 

106  Hrs. 

287  Hrs. 

1.25  Wks 

2.65  Wks 

7.175  Wks. 

.29  Mos. 

.61  Mos. 

1.65  Mos. 

Figure  VI-1.  ESTIMATES  OF  TIME  TO  PERFORM  ASSESSMENT  USING  ETAM  PROCEDURES 
(ASSUMES  AVAILABILITY  OF  AUTOMATED  PROCEDURES) 


VI-7 


TAEG  REPORT  NO.  12-3 


This  level  of  validation  awaits  the  future,  and  would  be  meaningful  not 
so  much  In  the  formal  exercise  as  in  the  nature  and  scale  of  adoption. 

ETAM  represents  systematic  rationality  on  a broad  scale,  and  in  many  en- 
vironments now,  would  be  a cultural  as  well  as  a technical  innovation. 
Cultural  acceptance  is  relatively  independent  of  technical  validity,  if 
indeed  the  two  forms  of  acceptance  are  distinct  from  each  other.  It  should 
be  conceded  that  introducing  ETAM  into  adoption  may  require  inventions  for 
cultural  acceptance  at  least  as  substantial  as  the  technical  inventions 
required  for  Its  creation. 

Many  of  the  recommendations  for  follow-on  efforts  are  directed  towards 
cultural  acceptance  in  the  milieu  of  its  intended  use.  Because  it  is 
completely  explicit  as  a decision-making  tool,  its  inadequacies  can  be 
identified  and  remedied  with  usage.  As  a structure  for  retaining  archival 
data,  it  also  enables  the  quality  of  decisions  to  be  assessed  and  thus 
potentially  Improved.  Because  of  its  scope,  decisions  in  training  and 
education  can  become  coordinated  rather  than  competitive.  This  would 
be  an  ultimate  validation. 
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This  appendix  includes  extended  descriptions  of  the  task 
structure  elements  and  their  training  and  learning  implications. 
It  is  intended  to  complement  the  material  in  Task  5 of  the  ETAM 
Procedures  (Section  III). 
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TASK  FORMATS  IN  HUMAN  PERFORMANCE 


Introduction 


Travelers  from  the  United  States  to  the  United  Kingdom,  who  overcome  their 
initial  fears,  may  be  pleasantly  surprised  to  discover  how  quickly  they  can 
"reverse"  many  of  their  driving  habits.  The  British  gearshift  is  operated 
by  the  left  hand  rather  than  by  the  right  hand,  the  driver  sits  on  the  right 
side  of  the  car  rather  than  the  left,  looks  to  the  left  for  the  rear  view 
mirror  rather  than  to  his  right,  and  drives  on  the  left  side  of  the  road 
rather  than  the  right.  Many  of  these  reversals  have  to  be  performed  simul- 
taneously such  as  when  making  turns  at  intersections.  The  need  to  reverse 
scanning  patterns  for  traffic  liabilities  still  further  complicates  the 
adaptation.  Yet  in  a relatively  few  hours,  depending  on  survival  during  the 
first  few  minutes,  the  driver  begins  to  be  comfortable  and  can  even  divert 
his  attention  to  viewing  the  countryside.  The  adaptation  is  made  in  perhaps 
a thousandth  of  one  percent  of  the  time  required  to  learn  the  original  driving 
skill.  There  is,  of  course,  large  individual  variation  in  such  adaptability, 
but  this  variation  may  be  largely  compounded  of  attitudes  towards  making  the 
change  rather  than  "information-processing"  capabilities. 

If  we  think  of  the  driver's  skill  only  as  a set  of  required  fixed  linkages 
between  what  he  sees  and  what  his  hands  and  feet  do,  it  is  impossible  to 
account  for  this  rapid  adaptation,  although  it  does  help  account  for  some  of 
the  errors  he  will  tend  to  make.  In  addition  to,  and  partly  independent  of, 
these  eye-hand  linkages  he  has  acquired  a model  or  "format"  for  receiving  and 
processing  classes  of  information  that  relate  (a)  his  purpose  of  the  moment, 

(b)  his  projection  of  the  track  of  the  vehicle  into  the  roadway  "corridor" 
ahead,  and  (c)  the  control  variables  of  direction  (steering),  velocity  (accel- 
erator and  brake)  and  power  ratio  (gearshift  positions).  The  input  variables 
and  the  output  variables  in  driving  remain  the  same,  although  the  data  content 
is  very  different!  It  is  this  processing  format  which,  when  established  by 
learning  a task,  can  mediate  substantial  shifts  of  learned  patterns  of  be- 
havior from  one  context  to  another. 

When  a brilliant  marketing  executive  is  hired  away  from  a soap  company  to 
organize  a marketing  operation  for  a clothing  firm,  his  knowledge  about  soap 
and  soap  markets  had  better  be  left  behind;  what  will  be  useful  will  be  his 
conceptual  models  and  formats  about  market  development  strategy  that  are  at 
least  partly  independent  of  the  product.  The  ability  of  the  experienced  pro- 
grammer to  learn  new  programming  languages  quickly  is  at  least  partly  mediated 
by  the  constants  in  program  language  structure  and  entities. 

The. design  of  training  and  the  design  of  task  tools  and  supports  can  be  enhanced 
by  a workable  awareness  of  how  formatting  occurs  in  the  learning  process,  and 
the  various  adaptive  functions  it  supports  in  the  economics  of  behavior. 

Definition 


A distinction  should  be  made  between  the  description  of  a task  format  and  the 
embodiment  of  a task  format  in  task  activities.  The  embodiment  of  a task  format 
in  human  performance  consists  of  situational  variables  and  the  response 
variables  that  recur  when  the  task  is  performed  many  times.  The  value  assumed 
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by  each  variable  may  change  on  each  occasion  the  task  is  performed.  Thus,  in 
driving  the  car,  the  picture  of  the  projected  track  or  corridor  for  the  auto- 
mobile through  traffic  and  the  transient  states  of  velocity  and  direction  are 
never  twice  the  same.  But  the  operational  variables  as  variables,  are  con- 
stantly present  each  time  the  task  is  performed. 

The  format  of  filling  out  a purchase  requisition  is  explicit  in  terms  of  the 
information  variables  printed  on  the  form.  The  requester  of  the  service  -- 
making  the  purchase  requested  by  the  requisition  --  is  given  the  standardized 
data  names  that  will  yield  the  information  essential  to  providing  the  service. 

The  printed  form  is  a cultural  invention  which  acknowledges  a "task  format"  in 
the  sense  described  here.  Despite  complaints,  the  printed  form  is  one  of  the 
more  powerful  labor-saving  devices  for  handling  transactions  among  organiza- 
tions that  request  and  supply  services. 

The  human  is  capable  of  learning  a large  variety  of  tasks  having  a variety  of 
structures:  perceptual,  interpretive,  decision-making,  constructing,  per- 

ceptual-motor. The  variety  of  tasks  is  somewhat  paralleled  by  variety  in  task 
formats:  we  shall  be  able  to  see  similarities  among  tasks  that  superficially 
appear  to  differ,  and  these  similarities  will  help  account  for  the  human  operator, 
having  learned  one  of  these  tasks,  to  learn  quickly  the  other  members  in  the 
task  group.  It  would  indeed  be  useful  to  know  if  perhaps  a few  dozen  task 
format  "archetypes"  might  account  for  sixty  percent  to  eighty  percent  of  work 
tasks  in  our  culture. 

The  reader  is  again  reminded  that  a task  format  can  be  thought  about  with  either 
of  two  different  references  in  mind.  From  one  point  of  view,  the  reference  is 
to  an  external  structure  --  such  as  a printed  transactional  form  --  while  from 
the  second  point  of  view,  a task  format  is  part  of  the  behavioral  pattern  that 
is  learned  by  an  operator  in  the  acquisition  of  a skill  or  ability.  Both  mean- 
ings of  task  format  have  utility  for  the  intelligent  design  of  task  supports 
and  training,  and  for  expectations  of  transfer  of  training  by  the  operator  from 
one  set  of  work  requirements  to  another.  But  unless  qualified,  the  expression 
"task  format"  in  this  paper  will  refer  to  a learned  pattern  of  organization  in 
behavior  rather  than  an  external  representation  of  a format. 

The  Learning  of  Task  Formats 

When  regularities  are  repeated  in  human  experience,  they  become  identified  and 
anticipated.  This  process  whereby  general  patterns  are  abstracted  from  the 
mass  of  variations  occurring  on  repeated  occasions  takes  place  spontaneously 
in  learning  the  task.  Thus,  as  an  individual  user  of  a computer  terminal  I 
have  learned  that  the  following  things  have  to  happen  when  I use  the  facility: 

The  device  must  be  turned  on. 

I must  identify  myself  with  an  acceptable  code. 

I may  have  to  respond  to  an  authorization  request. 

I must  identify  the  program  context  I plan  to  use. 

I may  have  to  cope  with  the  system  denying  me  service  because  it  is  too  busy. 

I must  request,  by  suitable  identification,  work  that  I have  stored  in  the 

system  on  previous  occasions,  or  state  that  I am  engaging  in  new  work. 
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In  some  applications  several  of  these  operations  may  be  procedural ly  merged, 
but  I am  aware  that  the  variables  nevertheless  exist.  If  I am  to  learn  to 
sign  on  and  use  a new  application,  or  a new  terminal -computer  configuration, 

I spontaneously  ask  for  the  specifics  of  procedure  on  each  of  these  variables. 
This  t'ormat  not  only  structures  my  getting  information  about  a modified  task 
"content:,  it  also  helps  me  remember  it  and  apply  what  I have  learned.  The 
model  may  also  help  me  interpret  why  something  goes  wrong  when  it  does.  Even 
in  a new  set  of  circumstances,  I can  generate  a set  of  "diagnostics"  for 
probing  into  a difficulty  on  the  basis  of  this  conceptual  structure  of  vari- 
ables in  this  class  of  activities.  Within  limits,  this  format  enables  me  more 
or  less  readily  to  substitute  one  response  mode  for  another:  pointing  at  menu 

selections  with  a light  pen  or  entering  the  data  from  a keyboard,  or  calling  it 
over  a telephone. 

The  greater  the  regularities  in  the  presence  of  the  same  variables  from  one 
task  occasion  to  the  next,  the  more  rapidly  will  implicit  format  develop. 
Regularities  consist  in:  the  same  set  of  variables;  the  same  sequence  in  time; 

the  same  arrangement  in  spatial  location.  If  the  sequence  in  which  the  operator 
must  cope  with  a set  of  task  variables  changes  from  one  occasion  to  the  next  -- 
that  is,  the  sequence  is  variable  or  random  --  the  task  format  will  be  learned 
and  be  a part  of  performance  somewhat  more  slowly.  That  is,  more  repetition  of 
learning  trials  will  be  necessary.  The  same  will  be  true  of  spatial  position 
of  the  task  variables  in  the  field  of  regard  or  in  the  work  environment.  Where 
there  is  standardization  both  of  sequence  in  the  operator's  coping  with  given 
task  variables,  and  in  their  absolute  or  relative  spatial  location,  the 
redundancy  increases  the  rate  at  which  the  task  format  becomes  learned  as  a 
factor  in  performance  proficiency.  It  is  a fundamental  principle  of  learning 
that,  up  to  some  point  of  diminishing  returns,  redundancy  of  task  cues  increases 
the  rate  of  learning  and  favors  retention  and  reinstatement  of  the  skill  over 
periods  of  disuse.  The  principle  is  significant,  even  though  in  highly  practiced 
performance  the  operator  gradually  tends  to  disregard  redundant  cues  by  selecting 
(through  spontaneous  processes)  a cue  element  that  is  sufficient  for  selecting  a 
task  response. 

To  the  extent  that,  in  early  learning  of  a task,  the  learner  engages  in  activ- 
ities during  practice  that  are  irrelevant  or  extraneous  to  the  processing 
format  to  be  developed,  the  format  will  be  learned  slowly.  For  example,  trouble- 
shooting electronic  equipment  can  be  divided  into  two  major  classes  of  activity: 
(1)  the  procedures  of  setting  up  test  equipment,  applying  probes,  taking  measure- 
ments, comparing  test  results  with  norm  readings,  searching  in  manuals  of  in- 
struction and  (2)  the  application  of  strategy  in  the  selection  of  that  next  test 
to  make  that  will  yield  the  largest  amount  of  diagnostic  information,  and  the 
logical  operations  of  deducing  the  significance  of  a given  test  finding  as  to 
what,  sections  of  the  system  may  be  excluded  from  further  suspicion.  Where 
practice  exercises  in  early  stages  of  learning  mix  these  classes  of  activities, 
the  formation  of  strategic  and  logical  skills  may  be  very  slow  because  of  the 
confounding  and  attenuating  information  required  for  the  procedural  operations. 
Obviously,  the  exercise  of  the  strategy  in  the  real  job  demands  integration  of 
the  activities,  but  the  formation  of  the  format  structure  may  require  at  least 
temporary  separation  during  training  of  these  two  sets  of  activity.  The  analysis 
of  the  psychological  functions  of  task  formats  in  a later  section  will  reveal 
the  justification  for  this  separation.  (It  is  possible  that  a task  format  can 
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be  so  overlearned  in  artificial  conditions,  that  it  may  not  adapt  to  combining 
with  other  formats  which  may  need  to  be  exercised  and  integrated  with  the 
first:  stage  of  learning  is  an  important  dimension  for  the  design  of  efficient 

training  schedules.) 

The  "Description  of  Formats"  and  "Behavioral  Formats" 

It  is  possible  to  memorize  the  bidding  rules  of  the  game  of  bridge  as  a set  of 
verbal  instructions.  If  they  can  be  recalled,  these  verbalized  rules  may 
assist  in  making  appropriate  bids,  but  in  themselves  they  are  insufficient  for 
excellence  in  bidding  --  even  if  the  psychology  of  bidding  were  to  be  set  aside. 
When  the  rules  are  embedded  in  the  skill  of  sizing  up  a hand,  the  bidder  can 
cope  with  exceptions  to  the  formal  rules,  making  tradeoffs  and  treating  the 
elements  in  his  hand  as  a pattern.  He  may  add  to  the  mere  bid  count  in  his  hand 
an  imaginary  playing  out  of  his  hand  against  his  opponents,  and  add  the  imagined 
outcome  to  his  assessment  of  the  bid  value.  The  verbalized  description  of  card 
value  count  making  up  a bid  serves  as  an  external  guide  that  should  help  him  to 
learn  the  format  as  a skill  component.  In  the  sense  that  it  is  a guide  to 
behavior,  it  is  of  course  a format  by  definition. 

Take  the  example  of  an  expense  account  form.  The  variables  on  the  form  specify 
the  kinds  of  information  that  ordinarily  are  necessary  and  sufficient  for  sub- 
mitting a statement  of  trip  expenses  according  to  the  requirements  for  accounting 
and  reimbursement.  It  is  the  external  format  for  generating  a statement  of  trip 
expenses.  It  may  be  contrasted  with  submitting  a statement  in  the  form  of  an 
essay.  The  printed  format  usually  helps  the  person  learn  comparatively  quickly 
a psychological  or  behavioral  format  for  selecting  data  from  perhaps  a random 
assortment  of  notes  on  expenses,  and  to  quickly  translate  them  into  entries  on 
the  paper.  The  implicit  format  for  expense  accounting  in  the  employee's  head 
will  also  tend  to  guide  him  during  his  trip  as  to  what  expenses  he  makes  nota- 
tions of,  collects  receipts  for,  remembers  and  qualifies.  The  accounting 
format,  when  in  his  head,  guides  his  spending  — or  at  least  his  attention  while 
spending.  The  format  becomes  operationally  meaningful  beyond  its  printed 
embodiment. 

The  point  here  is  that  it  is  naive  to  assume  that  because  a learner  has  been 
told  or  shown  a format  structure,  it  will  thenceforth  serve  as  an  effective 
component  in  his  skill.  Formats  become  learned  and  effective  in  skilled  per- 
formance as  one  learns  to  ride  a bicycle:  by  repeated  practice. 

The  Behavioral  Value  of  Learned  Formats 


Recall  that,  in  general  terms,  a task  format  is  a framework  of  those  factors  and 
variables  in  information  and  response  that  tend  to  be  constant  in  many  repeated 
cycles  of  the  task.  When  a format  becomes  learned  as  a component  structure  in 
a task  skill,  it  serves  as  a behavior  organizer  in  the  following  ways. 

1.  Directing  attention.  The  task  format  embodies  selective  attention.  The 
operator  learns  what  variables  or  properties  of  the  work  environment  are  task- 
relevant, and  those  that  are  irrelevant  to  any  task  action.  When  the  operator 
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has  learned  what  specifically  to  look  for  as  essential  work  cues,  and  can 
disregard  the  non-essential  and  irrelevant,  the  burden  on  his  attention  and 
information  processing  filters  is  substantially  reduced.  It  may  again  be 
emphasized  that  the  format  directs  attention  to  categories  of  task  cues, 

"stimulus  variables"  or  "task  parameters"  — where  each  of  these  expressions 
is  intended  to  mean  about  the  same  thing.  The  effect  of  the  format  is,  there- 
fore, both  that  of  a selective  filter  and  that  of  a scanner  for  what  have  been 
learned  as  "essential"  categories  of  task  data.  When  attention  becomes  auto- 
matically directed  and  selective,  the  further  processing  of  task  data  can  be- 
come simplified. 

Efficiency  is,  however,  bought  at  a price  and  the  price  usually  Is  some  degree 
of  rigidity.  The  task  format  can  be  practiced  by  repetitive  experience  to  the 
point  that  the  operator  disregards  — fails  to  notice  — what  should  be 
qualifying  conditions,  or  out-of-the-ordinary  demands.  His  perceptions  can  be- 
come stereotyped  so  that  he  operates  on  the  minimum  standard  format.  His  per- 
formance now  becomes  vulnerable  to  a demand  for  behavior  that  requires  accepting 
and  processing  information  that  is  outside  his  format  structure. 

The  determination  of  what  are  the  "essential"  cue  variables  in  a task  does  not 
arise  by  magic  from  some  abstract  specification  of  ideal  task  performance.  It 
arises  from  the  psychological  operations  by  which  input  states,  goal  concepts 
and  output  actions  are  linked  to  consequences  as  perceived  by  the  learner  over 
a substantial  number  of  learning  and  performance  trials.  (All  performance  has 
some  learning  effect.)  By  the  general  principle  of  least  effort,  continued 
practice  will  result  in  the  processing  of  the  least  amount  of  information  that 
is  sufficient  to  produce  results  that  are  adequate  or  "satisfy"  the  performer  — 
or  do  not  interfere  with  his  satisfactions,  which  is  perhaps  a different  picture. 
Understanding  this  principle  should  help  in  the  programming  of  learning 
experiences. 

There  is  irony  in  the  development  of  perceptual  rigidity.  As  the  learner  acquires 
perceptual  formats,  the  demands  on  his  attention  dealing  with  standard  task  data 
tend  to  become  less,  so  that  in  effect  he  can  disperse  attention  over  a larger 
field  of  situational  content.  He,  therefore,  is  acquiring  the  capacity  for  more 
flexible  perception,  both  in  range  and  detail,  but  he  must  be  induced  to  exercise 
this  channel  capacity.  If  he  must  learn  to  perform  several  streams  of  task 
activities  more  or  less  concurrently,  this  is  in  part  how  this  added  channel 
capacity  is  acquired. 

It  is  possible  for  the  enlightened  learner  to  continue  to  impose  on  himself 
continuously  more  rigorous  criteria  of  attention  to  nuances  in  input  information 
after  he  has  mastered  the  ability  to  process  the  minimum  required  data  in  a 
task  --  that  is,  he  uses  the  task  format  as  a performance  tool  rather  than  being 
used  by  It  as  a tyrannical  constraint  and  limit.  This  assertion  assumes  some 
cognitive  freedom  of  choice  rather  than  the  notion  that  learning  must  be  acquired 
only  through  external  "shaping"  of  behavior  at  the  time  of  skill  acquisition. 

There  are,  no  doubt,  large  individual  differences,  perhaps  arising  from  many 
causes,  in  this  self-instructing  phenomenon. 
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In  conclusion,  the  learning  of  a task  format  has  a powerful  effect  in  restricting 
the  range  of  what  the  operator  pays  attention  to  and  notices  in  the  total 
stimulus  environment  of  the  task.  This  restriction  and  focusing  of  attention 
is  one  factor  in  the  improvement  of  qualitative  and  quantitative  factors  in  task 
performance.  The  greater  the  standardization  of  information  categories,  and  the 
temporal  and  spatial  presentation  of  given  categories  of  task  information,  the 
more  quickly  will  the  format  be  learned  as  manifest  in  selective  attention.  If 
the  task  demands  sampling  for  periodic  changes  in  value  of  given  task  input 
variables,  the  standardization  of  spatial  location  of  the  indicator  for  the 
variable  assumes  special  significance  when  the  operator  is  highly  loaded  or 
stressed. 

2.  Organizing  short  term  memory.  Short  term  memory  --  or  "working  memory"  — 
is  the  retention  of  variable  information  peculiar  to  a given  cycle  of  task  per- 
formance. It  ranges  from  keeping  in  mind  one's  "place"  in  a procedural  sequence, 
or  perhaps  briefing  instructions  for  a particular  set  of  job  operations,  to  the 
many  factors  of  demand  and  resource  and  sequence  in  formulating  the  solution  to 
a complex  logistics  problem.  For  practical  purposes,  the  complexity  of  the 
problem  which  the  human  can  solve  is  limited  by  the  amount  of  information  he  can 
effectively  hold  and  use  in  his  short  term  memory.  Short  term  memory  can,  of 
course,  be  assisted  by  job  aids  which,  like  a displayed  map,  may  refresh  the 
content  in  short  term  memory  but  cannot  serve  as  Its  equivalent. 

At  any  given  moment  in  a work  cycle  short  term  memory  tends  to  hold  representa- 
tions of  task  conditions  and  status,  momentary  purpose,  and  perhaps  a symbolic 
key  to  the  invoking  of  a class  of  response  operations  to  be  performed  on  the  data. 
Where  the  operator's  experience  has  encountered  a high  degree  of  regularity  in 
time,  space  and  data  content  of  task  stimulus  and  task  response,  the  content  of 
short  term  memory  tends  to  become  reduced  to  merely  that  situational  data  suffi- 
cient to  evoke  the  next  response  of  execution.  To  the  extent  that  a stimulus 
sequence  is  invariant,  the  response  sequence  may  be  invoked  and  executed  without 
attention  to  the  elements  within  the  stimulus  sequence,  but  as  an  automatic 
train  of  response. 


Since  performance  is  limited  by  the  ability  of  short  term  memory  to  cope  with 
task  information,  both  the  design  of  the  task  and  training  for  the  task  should 
be  directed  towards  the  structuring  of  short  term  memory  content.  This  structure 
is  based  upon  the  task  variables  which  make  up  the  situational  context  to  which 
task  responses  are  made. 

With  practice  on  a task,  the  learning  processes  seem  to  sort  out  the  recurring 
task  variables  so  that  short  term  memory  develops  the  equivalent  of  a matrix 
or  template  for  the  reception  and  implicit  "labeling"  of  classes  of  task  data. 

A limited  (and  partially  misleading)  analogy  is  that  of  setting  up  a set  of 
standard  pigeon-hole  locations  for  task  data  variables.  The  analogy  breaks 
down  because  there  is  interaction  among  the  contents  of  pigeon-holes:  pattern 

as  well  as  elements  may  be  perceived  by  the  operator  --  the  content  may  operate 
dynamically  rather  than  passively  like  Impulses  written  into  the  register 
positions  of  a computer.  Thus,  the  driver  of  the  car  may  implicitly  note  his 
own  speed  and  direction  and  the  speed  and  direction  of  another  car,  and  more 
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or  less  continuously  compute  the  projections  of  the  data  samples  with  respect 
to  collision  probabilities.  Or  the  symptoms  reported  by  the  patient  to  the 
diagnostician  may  continuously  summate  in  the  physician's  mind  into  a pattern 
which  he  recognizes  as  a disease  entity. 

The  value  to  performance  of  this  parameterizing  of  task  data  is  clearly 
enormous.  That  task  data  which  demands  differential  task  response  is  sorted 
out  from  that  which  is  either  irrelevant  to  the  moment,  or  calls  for  a 
standardized  subroutine  of  pre-established  responses.  The  psychological 
categorization  of  data  enables  it  to  be  apprehended  and  stored  both  as  data 
elements  in  the  matrix,  and  as  a pattern  to  be  completed.  In  an  operational 
sense,  short  term  memory  has  become  "organized."  Increased  practice  on  the 
task  increases  the  stability  of  this  organization  of  input  data  structuring. 

The  result  is  an  effective  or  functional  increase  in  the  capacity  of  short 
term  memory.  That  is,  to  the  extent  that  existing  data  patterns  are  well 
structured  in  short  term  memory,  additional  data  can  be  stored  in  it.  The 
driver  with  increasing  competence  can  note  and  remember  additional  data  in  the 
driving  environment;  the  diagnostician  can  hold  in  mind  a larger  set  of  data 
specific  to  this  patient  and  his  ailment;  the  architect  designer  can  optimize 
a larger  set  of  design  parameters  and  constraints  in  the  course  of  design. 

A major  function  for  task  performance  served  by  short  term  memory  is  the 
capacity  to  separate  stimulus  information  temporal 1y  from  response  execution. 

Thus,  the  accomplished  typist  apprehends  a group  of  letters  and  words  as  a 
clump  which  is  stored  in  short  term  memory  which  is  later  executed  as  a clump 
by  her  fingers  on  the  keyboard.  A contrasting  organization  of  work  would  consist 
of  looking  at  an  input  character  in  the  source  document  and  keying  that  character, 
then  looking  at  the  next  character  and  keying  it.  The  temporal  separation  of  in- 
put from  output  enables  the  operator  to  prepare  adjustments  to  momentary  complexi- 
ties of  response  demand  without  a disturbance  to  the  pace  of  the  output.  Vari- 
ations both  in  the  complexity  in  the  input  and  complexity  in  output  response 
tend  to  be  smoothed  out  by  some  separation  of  rate  in  the  reception  of  stimulus 
input  and  execution  of  output.  A pattern  of  input  is  apprehended  and  briefly 
stored  and  matched  to  a pattern  of  output.  All  of  these  factors  together  reduce 
fatigue  in  the  performance  of  the  task,  and  delay  the  onset  of  "blocks"  --  a 
period  of  time  when  within  the  operator,  presumably  from  fatigue,  there  is  a 
psychological  block  between  input  processing  and  output  processing.  These  are 
also  occasions  for  errors. 

To  the  extent  that  the  rate  of  apprehending  stimulus  input  is  faster  than  the 
rate  of  response  execution,  the  operator  can  attend,  at  least  intermittently, 
to  other  streams  of  information.  For  example,  he  may  monitor  the  outcomes  of 
executed  responses. 

Short  term  memory  mechanisms  can  be  viewed,  therefore,  as  information  buffers 
that  enable  temporal  flexibility  between  the  presentation  of  a task  stimulus 
and  its  action  in  eliciting  a task  response.  This  flexibility  exists  as  a 
capacity.  How  this  capacity  becomes  an  operational  ability  depends  largely  on 
the  conditions  and  extent  of  task  practice  from  which  no  skill  can  be  inde- 
pendent. The  operator  learns  what  he  does.  What  he  does  may  be  modified  both 
by  external  circumstances  and  by  what  he  tries  to  do  by  voluntary  processes. 
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The  buffer  in  short  term  memory  may  be  thought  of  as  consisting  of  a set  of 
registers.  Each  register  position  has  become  specialized  to  receive  the  data 
value  of  a given  task  variable.  When  the  registers  in  the  buffer  are  filled, 
the  content  as  a pattern  is  dumped  into  an  action  converter.  This  mechanistic 
description  is  Intended  only  as  a highly  oversimplified  analogy  and  should  in 
no  way  be  treated  literally.  The  point  to  be  understood  is  the  potential  for 
increased  processing  efficiency  when  processes  of  this  kind  result  in  the 
organization  of  short  term  memory  content. 

3.  Task  closure  and  reset.  The  learned  structures  that  operate  in  short  term 
memory  may  also  yield  the  phenomenon  of  the  sense  of  task  completion.  This  * 
source  of  completion  may  be  accompanied  by  the  erasure  of  much  of  the  variable 
content  in  short  term  memory.  Thus,  one  looks  up  a telephone  number,  and  per- 
haps keeps  it  in  mind  until  the  digits  have  been  dialed.  On  the  completion  of 
the  dialing,  the  number  may  tend  to  be  forgotten  at  once.  The  "space"  in  short 
term  memory  is  now  available  for  other  semi -transient  data.  And  the  human  has 
an  accompanying  sense  of  completion:  the  task  or  sub-task  is  "closed". 

What  constitutes  a task  closure  depends  on  how  the  task  is  learned.  When  a 
telephone  operator  mediates  between  a caller  and  the  station  or  person  being 
called,  her  task  Is  ordinarily  not  completed  until  she  verifies  the  calling 
number.  She  will,  therefore,  tend  to  retain  the  calling  number  in  mind  (assuming 
it  hasn't  been  written  down)  until  it  has  been  verified.  When  it  has,  then  and 
not  until  then,  will  she  have  closure  and  the  unbuffering  of  the  transient 
content.  The  understanding  of  this  phenomenon  should  help  in  the  design  of  task 
training.  Thus,  if  a pilot  or  a maintenance  mechanic  has  extensively  practiced 
and  learned  his  primary  tasks  as  being  completed  without  filling  in  a debriefing 
report,  he  learns  closure  before  filing  the  resport  of  what  happened  during  the 
task.  If  now  he  must  file  the  report,  his  memory  of  task  data  may  be  unreliable 
and  incomplete.  But  if  filing  a terminal  report  is  learned  as  a normal  portion 
of  a work  cycle,  his  closure  will  be  delayed  until  completion  of  the  report. 

These  comments  must  be  qualified.  Closure,  as  we  know  from  everyday  experience, 
Is  not  all-or-none.  Insofar  as  forgetting  is  concerned,  it  is  a matter  of 
degree.  Since  tasks  are  composed  of  subtasks  and  also  of  higher  order  tasks, 
closures  are  also  hierarchic.  Furthermore,  there  are  relationships  between 
short  term  memory  and  long  term  memory  whereby  some  content  in  the  former  be- 
comes content  in  the  latter.  But  mechanisms  of  relationship  between  short  term 
memory  and  long  term  memory  are  outside  the  scope  of  this  paper.  Task  formats 
are  retained  in  long  term  memory  from  which  they  are  accessed  and  enter  opera- 
tionally into  short  term  memory  under  cues  that  initiate  a given  task  activity. 

A person  is  "set"  for  a task  or  a task  cycle  when  the  task  format  has  entered 
short  term  memory  from  long  term  memory. 

A final  practical  comment.  The  learning  and  performance  situation  of  the  learner 
can  be  such  as  to  foster  the  sense  of  closure  when  he  has  completed  some  ter- 
minating task  response.  If  so,  he  will  tend  not  to  search  for  evidence  of  out- 
come adequacy  of  his  performance,  nor  to  link  efficiently  (in  the  psychological 
sense)  such  outcomes  with  the  elements  of  task  situation  and  task  response. 

This  pattern  of  behavior  will  tend  to  preclude  the  progressive  improvement  of 
his  own  performance  with  respect  to  given  criteria  except  by  externally  imposed 
instruction.  The  most  effective  context  for  learning  occurs  when  the  information 
of  the  task  cycle  is  still  in  mind  and  the  resulting  outcome  is  presented  while 
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that  Information  is  still  "live"  in  present  memory,  which  must  be  preliminary 
to  closure  as  described  above.  This  observation  suggests  not  merely  a 
temporal  connection  between  performance  of  task  and  presentation  of  outcome: 
it  also  points  to  what  the  operator  has  learned  as  the  completion  status  of  a 
task  v.ycle.  These  factors  suggest  why  so  much  academic  study  aimed  at  "finish- 
ing the  test"  is  not  only  inefficient  but  poorly  structured  for  enabling  the 
student  to  become  an  effective  self-teacher. 

4.  Transfer  of  skill.  The  task  format,  like  a template  for  inserting  the 
values  of  variables  in  a more  or  less  standard  equation,  enables  large  vari- 
ations in  the  values  of  the  situation  variables  to  be  processed  effectively  by 
the  operator.  The  format  may  thus  play  a major  role  in  enabling  rather  sub- 
stantial changes  in  task  content  and  task  setting  to  be  learned  quickly, 
relative  to  a control  condition  where  the  new  task  was  learned  without  benefit 
of  existing  format. 

Perhaps  it  should  be  underscored  that  a highly  routinized  task  that  Is  always 
performed  In  an  invariant  way  to  an  invariant  sequence  of  readily  identified 
stimulus  elements  may,  when  practiced  many  times,  be  performed  purely  as  a 
rote  sequency  with  a minimum  development  of  a transferable  format  or  abstract 
or  implicit  model  of  the  task.  Formats  develop  to  cope  with  variability  within 
the  task  and  from  one  task  occasion  to  the  next.  In  other  words,  where  the  task 
"variables"  assume  only  a single  value,  and  little  or  no  information  is  carried 
over  from  previous  steps  to  later  steps  in  a sequence  of  actions,  the  procedure 
will  tend  to  become  automatized.  Little  transfer  is  to  be  expected,  unless  the 
operator  learns  a variety  of  tasks  having  the  same  structure.  In  this  case, 
the  conditions  of  variability  are  met,  and  a task  format  will  tend  to  develop. 
The  result  will  aid  progressively  in  still  more  new  tasks  having  similarities 
in  structure.  For  example,  a shop  man  learns  and  practices  for  years  taking 
the  same  kind  of  bicycle  apart  and  putting  it  together.  He  will  be  inept  in 
learning  to  disassemble  and  assemble  mechanical  objects  different  from  that 
bicycle  --  say,  a gasoline  engine.  But  a person  who  has  taken  apart  and  put 
together  a variety  of  objects  learns  a general  pattern,  including  some  strat- 
egies, for  this  class  of  work.  He  can,  with  comparatively  little  guidance, 
disassemble  and  assemble  devices  quite  strange  to  him.  He  has  acquired  a 
general  format  for  disassembling  and  assembling  devices. 

There  are,  of  course,  factors  other  than  task  formats  which  can  be  responsible 
for  transfer  of  training:  a common  base  of  nomenclature  and  reference  informa- 

tion between  the  old  and  new  task  are  examples. 

Types  of  Task  Format 

Task  formats  are  as  numerous  as  there  are  classes  of  tasks.  This  writer,  in 
several  reports  (1969,  1973),  has  analyzed  several  dozen  that  might  be  regarded 
as  widely  distributed  in  the  domain  of  cognitive  and  perceptual -motor  work. 

There  are  a few  which  may  be  regarded  as  archetypical. 

Verbalized  self-instructions  and  external  formats.  Task  performance  can  be 
structured  by  verbal  instructions  given  to  oneself  or  given  by  a manual  or  an 
instructor.  The  information  In  the  verbalized  instruction  may  contain  informa- 
tion necessary  for  actual  performance,  but  not  sufficient.  There  must  be  a 
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capability  for  translating  the  verbal  instruction  into  perceptual  actions  and 
muscular  or  other  output  actions.  The  self-instruction  is  an  external  format 
for  performing  the  task:  it  guides  behavior  without  in  itself  being  that 

behavior. 

A printed  form  to  be  followed  in  a transaction  is  another  example  of  an 
external  format.  It  specifies  the  classes  (variables)  of  information  necessary 
to  obtain  in  order  that  a given  action  can  be  taken.  The  printed  form  not  only 
facilitates  the  performance  of  the  task  of  getting  and  setting  down  the  essential 
data,  it  also  simplifies  the  learning  of  that  task  as  contrasted  with  a situation 
In  which  such  a guide  is  absent.  It  enables  the  operator  to  restrict  his  opera- 
tions in  getting  the  data  content:  he  does  not  have  to  think  about  what  kinds 

of  data  to  provide.  (Obviously,  efficiency  here  has  Its  tradeoff  in  flexibility 
and  adaptiveness,  but  we  cannot  address  all  Issues  at  once!) 

There  may  be  procedure  formats.  An  example  may  be  that  of  formulating  a search 
specification  into  a data  base  system.  The  operator  may  have  the  task  of 
searching  a large  data  base  of  electric  motors  and  parts  by  attributes  and 
functions  associated  with  given  parts.  This  requires  him  to  name  attributes 
and  the  associated  attribute  value  as  well  as  to  Indicate  logical  connectors 
such  as  AND,  OR  or  NOT  between  elements  of  the  search  request.  A graphic  dis- 
play may  show  him  boxes  and  box  names  to  fill  in  --  a standard  format  for 
creating  the  specification  of  his  inquiry:  he  may  or  may  not  have  to  follow  a 
prescribed  sequence  for  entering  the  data.  This  is  an  external  task  format. 

He  may  have  to  learn  the  format  --  as  contrasted  to  using  a "free  form"  of 
inquiry  specification  — but  once  the  format  is  learned,  he  will  tend  sponta- 
neously to  think  of  his  request  in  this  format.  The  format  becomes  internalized. 
When  it  is,  it  will  take  less  of  his  channel  capacity  (and  probably  produce 
fewer  unintentional  errors)  than  if  he  were  to  continue  in  "free  form".  But 
even  allowing  him  free  form  mode  would,  with  repetitive  experiences,  result  In 
his  developing  (eventually)  his  internal  format  which  standardizes  the  opera- 
tion for  him.  Repeated  human  experiences  lead  to  regularities  in  behavior 
which  Is,  of  course,  a major  characterization  of  what  happens  in  learning  a 
task. 

Procedural  sequentialities.  If  an  individual  is  placed  in  a variety  of  situa- 
tions  all  of  which  have  a general  pattern,  he  will  tend  to  learn  and  anticipate 
the  elements  in  the  pattern.  Thus,  the  operation  of  a mechanical  vehicle 
generally  requires  getting  into  it,  turning  it  on,  engaging  power  to  the 
wheels,  releasing  a braking  mechanism  and  proceding.  Most  of  these  operations 
apply  equally  to  an  automobile,  aircraft  or  boat.  Depending  perhaps  partly  on 
intelligence,  but  more  on  the  variety  of  experiences,  this  pattern  of  sequen- 
tiality can  be  adapted  to  novel  situations. 

Operations  with  a weapon  have  the  general  sequence  of  preparation  for  use,  load- 
ing, aiming  and  firing.  It  is  not  necessary  that  the  Individual  be  able  to 
verbalize  the  format  in  order  to  have  it  expressed  in  his  behavior.  The  format 
is  expressed  in  behavior  by  the  relative  rate  at  which  he  can  learn  the  task 
as  compared  with  a control  subject  who  lacks  the  experiences  for  having  developed 
the  format.  Clearly,  a format  in  behavior  may  have  many  individual  levels  of 
specificity  and  generality.  No  sets  of  habits  are  alike  in  any  two  individuals, 
and  an  internal  format  is  a learned  entity  that,  in  psychological  jargon,  is  a 
habit. 
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Transactional  formats.  The  general  structure  of  a transactional  format  consists 
of  the  following  constituents:  (a)  a set  of  more  or  less  standardized  input 

variables  to  a processing  function;  (b)  a set  of  variables  of  reference  informa- 
tion that  supports  the  input  data  for  the  processing  function;  (c)  a set  of 
more  or  less  standardized  output  variables  from  the  processing  function;  and 
(d)  the  processing  function  itself.  Some  transaction  structures  may  include  a 
comparator  of  the  processing  output  with  an  idealized  or  criterion  output  and 
if  the  actual  output  is  out  of  prescribed  tolerance,  a feedback  arrangement 
leading  to  a recycle  of  the  input  with  the  benefit  of  the  added  corrective  data. 
In  the  case  of  interest  to  us  here,  the  processor  is  a person. 

An  Illustration  will  help  to  translate  this  definition  into  a clearer  picture. 

A dispatcher  takes  trouble  calls  and  selects  what  repairmen  will  take  what  calls 
and  in  what  order.  His  inquiry  about  the  trouble  will  tend  to  consist  (in  this 
particular  task)  of  a pattern  of  the  same  variables:  name  and  address  of  the 

caller  and  geographical  location  of  the  trouble,  the  urgency  of  the  trouble  and 
the  nature  of  the  trouble.  He  has  learned  to  extract  those  essential  classes 
of  data  that  help  him  decide  (a)  the  priority  to  assign  to  the  service  if  he 
has  a queue  of  requests;  (b)  whom  to  send;  (c)  what  supplies  and  tools  to  send; 

(d)  whether  to  send  only  one  or  more  than  one  man.  These  are  the  output  vari- 

ables in  his  decision-making.  His  reference  information  may  consist  of  the 
following:  (a)  the  implications  of  any  in  a given  set  of  reported  troubles  to 

the  customer;  (b)  the  implications  of  a given  class  of  trouble  for  the  company; 
(c)  the  relative  importance  of  a given  customer;  (d)  the  abilities  of  each  of 
his  service  crew;  (e)  the  availability  of  members  of  his  service  crew  with 
special  skills  who  could  respond  to  the  trouble  call.  He  attempts  a best  match 

between  the  variables  in  the  demand  for  service  and  the  properties  of  the  re- 

sources available  for  responding  to  the  demand.  His  mental  processes  leading 
to  the  decision  may  include  various  strategies  and  tradeoff  values. 

This  describes  decision  structure.  (It  also  is  a specification  for  a procedure 
to  develop  data  base  content  and  structure  for  the  support  of  human  decision- 
making tasks.)  These  variables,  more  or  less,  apply  to  any  dispatching  and 
scheduling  task.  The  task  format  that  is  learned  by  this  dispatcher  may  be 
transferred,  with  various  degrees  of  savings  in  learning,  to  a wide  variety  of 
contexts  from  telephone  repair  to  the  handling  of  contingencies  on  the  shop 
floor.  (The  greater  the  overlap  of  reference  knowledge  variables  between  the 
old  and  new  tasks,  the  greater  the  additional  savings  in  training.)  Obviously, 
the  extent  to  which  the  new  task  to  be  learned  has  explicit  parallels  in  its 
structure  of  variables  to  the  old  task,  the  more  rapidly  will  the  learner  adapt. 

This  transaction  structure  is  also  applicable  to  the  understanding  of  task 
formats  dealing  with  multiple-variable  tracking  tasks  such  as  operating 
vehicles.  This  form  of  analysis  is,  of  course,  precisely  the  approach  that  the 
engineer  and  computer  programmer  must  take  when  they  try  to  automate  a manual 
activity:  isolate  and  identify  the  input  variables  and  the  control  variables 

essential  to  the  task. 

Some  external  arrangements  of  the  task  make  implicit  formats  easier  to  learn 
and  reinstate,  and  generally  offer  more  flexibility  in  performance.  For 
example,  if  the  entire  set  of  task  variables  comprising  the  format  is  displayed 
to  the  operator  at  one  time,  like  a map,  the  pattern  of  variables  will  more 
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quickly  be  learned.  The  example  is  that  of  the  printed  expense  account  form, 
or  the  engineering  change  form.  The  alternative  arrangement,  fostered  by 
limitations  in  computer  display  technology,  is  to  display  only  one  variable 
at  a time  until  the  operator  responds  to  it  --  the  remainder  of  the  trans- 
action is  hidden  from  his  view.  The  same  principle  of  showing  the  entire 
structure  all  at  once,  rather  than  a fragment  at  a time,  is  probably  desirable 
for  training,  perhaps  even  where  the  operational  environment  may  present  only 
one  or  a few  variables  at  a time. 

Obviously  the  visual  layout  of  the  external  format  should  parallel  the  sequence 
in  which  information  is  received,  and  is  grouped  and  organized  for  reaching 
decisions.  If  reference  information  is  externally  provided  (that  is,  external 
to  the  operator's  head  such  as  a display  screen)  its  layout  in  terms  of  data 
names  and  attributes  of  entities  should  be  consistent  with  the  implied  demands 
of  input  variables,  output  variables  and  processing  strategy  in  the  decision- 
making operation. 

Even  though  a learned  task  format  can  mediate  transfer  of  training  across  sub- 
stantial variations  in  task  setting  and  task  content,  the  larger  the  number  of 
identities  between  stimulus  layout  and  response  selection,  the  more  rapidly 
can  the  operator  learn  the  second  task. 

Psychol ingui stic  Formats 

In  no  existing  field  of  behavioral  inquiry  has  the  construct  of  "task  format" 
been  so  widely  investigated  as  in  the  psychology  of  grammar  and  psycholigui sties 
the  study  of  structure  in  verbal  expression.  It  is  indeed  a fact  that  the 
"rules"  of  grammatical  constructions  become  embedded  in  our  verbal  behavior, 
although  the  rules  may  be  manifest  in  well-nigh  Infinite  variation  of  content. 
The  structure  of  "format"  of  the  sentence  as  a "unit  of  thought"  seems  funda- 
mental, as  are  the  sentence  structure  variables  of  subject,  predicate  and 
object,  and  the  association  of  qualifiers  (modifiers)  with  each  of  these  parts. 

It  is  outside  the  scope  of  this  paper  and  the  competence  of  the  writer  to  dis- 
course on  this  topic.  It  is  cited  as  a widely  examined  example  of  the  fact 
that  humans  develop  and  use  organizing  structures  of  variables,  and  relations 
among  variables,  in  behavior  that  operates  with  wide  varieties  of  content. 

In  this  context,  another  point  can  be  made  about  the  difference  between  an 
external  format  and  an  internal  format.  The  external  format  of  sentences  is 
expressed  by  written  rules  about  grammatical  "correctness".  It  should  be 
obvious  that  Individuals  do  not  literally  incorporate  these  rules  in  the  form 
of  these  verbal  expressions  in  the  formulation  of  their  sentences.  The  verbal 
expressions  of  rules  may  guide  the  selection  and  implicit  editing  of  sen- 
tences expressing  cognitive  content,  and  thus  assist  in  learning  to  criterion 
performance.  But  the  implicit  formats  for  the  "grammatical"  construction  of  a 
sentence  come  to  operate  at  behavioral  levels  where  the  structure  of  a verbal 
rule,  such  as  "one  should  not  split  Infinitives",  is  as  irrelevant  and  mis- 
leading as  to  say  that  the  mechanical  design  of  a switch  contains  the  "rule" 
whereby  It  will  open  or  shut.  If  this  kind  of  analogy  is  used  (and  some  of  us 
do  use  it),  the  context  should  be  clear. 
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One  learns  to  operate  with  behavioral  models  not  by  verbal  descriptions  of 
the  models  --  except  at  very  preliminary  stages  — but  by  the  exercise  of  the 
model  in  behavior.  It  is  not  clear  that  a large  body  of  educational  and  train- 
ing practice  recognizes  the  difference  --  as  well  as  the  relationships  --  be- 
tween external  task  formats  and  implicit  embodiments  in  behavior. 

For  the  time  being,  let  us  leave  psycholinguistic  models  with  the  scholars. 

They  have  given  us,  however,  an  example  of  the  notion  of  task  format  in  a 
respectable  field  of  inquiry.  Their  findings  about  the  processes  whereby 
implicit  cognitive  formats  develop  and  operate  may  help  us  better  understand 
these  phenomena  in  other  than  linguistic  behavior. 

"Understanding11  a Task 

Except  where  the  manager  must  serve  the  role  of  teacher,  it  is  generally  recog- 
nized that  "management"  of  workers  does  not  require  the  ability  to  perform 
their  various  tasks  at  acceptable  skill  levels.  The  unsettled  question  then 
becomes  "What  should  a manager  know  about  the  job-tasks  he  manages  in  order  to 
assign,  coordinate  and  evaluate?"  A reasonable  proposal  is  that  he  should  be 
able  to  identify  and  describe  the  formats  of  the  tasks  of  the  people  who  report 
to  him  — where  these  descriptions  take  the  structures  that  have  been  discussed 
in  previous  pages.  This  is  the  structure  of  the  task  variables  in  input  problem 
situation,  output  response  and  reference  entities  and  attributes.  The  details 
of  the  processing  rules  and  operations  that  translate  input  problem  to  output 
solution,  although  obviously  essential  to  performing  the  task  (and  to  teaching 
it)  are  secondary  to  a useful  understanding  of  it  for  management  purposes. 

If  a working  knowledge  is  to  be  extended  beyond  format  structure,  however,  the 
most  useful  information  about  the  processing  function  consists  of  the  strategic 
tradeoffs  in  the  task  criterion.  In  other  words,  what  is  the  task  operator  try- 
ing to  optimize  (or  should  optimize)?  The  dispatcher  was  trying  to  optimize 
the  matching  of  a resource  skill  with  a resource  demand  but  with  some  minimum 
of  delay  in  time- to- repair  multiplied  by  the  severity  of  the  problem.  Even  if 
objectively  measurable,  these  sets  of  parameters  are  not  linear  nor  entirely 
independent  of  each  other,  so  that  "optimization"  may,  under  heavy  load,  have 
to  be  substituted  by  a satisficing  criterion. 

If  the  manager  has  an  operational  awareness  of  (a)  the  task  format;  (b) 
criteria  of  effective  as  contrasted  with  ineffective  levels  of  performance  and 
(c)  the  strategic  tradeoff  variables  to  be  optimized  or  satisficed  in  perform- 
ing the  task,  he  will  often  have  more  explicit  knowledge  of  the  task  than  the 
incumbent  on  the  job. 

This  kind  of  understanding  can  be  functionally  useful  to  professional  persons 
whose  work  interfaces  with  other  expertise.  It  should  also  be  useful  to  those 
engaged  in  cross-training  of  personnel  for  career  guidance  and  in  the  struc- 
turing of  performance  tests.  Its  significance  to  those  engaged  in  the  design 
of  tasks  and  task  supports  has  been  mentioned. 

This  kind  of  information  about  tasks  may  be  contrasted  with  job-task  descrip- 
tions consisting  of  a list  of  work  "functions"  or  of  detailed  procedural 
statements. 
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Conclusion 


Learning  that  is  directed  towards  performance  criteria  is  a process  of  increas- 
ing order  and  organization  from  states  of  disorder  and  uncertainty.  The  ex- 
pression of  disorder  is  contained  in  the  statement,  "What  am  I suppoed  to  do 
and  how  do  I do  it?"  The  level  of  a competence  or  skill  is  tested  by  the 
operator's  coping  with  circumstances  that  he  has  not  precisely  encountered  be- 
fore: the  imposition  of  order  on  uncertainty  and  variability. 

Depending  on  the  nature  of  the  task,  the  conditions  of  training  and  performance 
and  perhaps  on  the  cognitive  capabilities  of  the  operator  — learning  and  per- 
formance may  occur  at  more  than  one  apparent  level.  To  the  extent  that  there 
is  variation  in  conditions  between  one  cycle  of  performing  the  task  and  another, 
the  operator  learns  not  merely  a one-for-one  stimulus-response  relationship. 

He  acquires  a format  of  processing  variables.  A mathematical  equation  is  an 
extremely  crude  and  unsophisticated  analogy:  the  value  of  variable  x added  to 

the  value  of  variable  y yields  the  value  of  variable  z:  that  is,  "Add  the  rate 

of  the  ro wer  downstream  to  the  rate  of  the  current  of  the  stream  to  determine 
the  rate  at  which  the  boat  is  moving  downstream."  The  process  of  learning  — 
at  least  under  some  conditions  — enables  subjective  "formulas"  to  be  developed. 
The  requirement  is  some  pattern  of  regularity  in  the  variables  of  the  problem 
presented  to  the  learner,  but  variability  in  the  values  that  the  variable  take. 
The  clod  does  learn  the  physics  of  operating  a wheelbarrow  over  rough  terrain 
in  the  sense  that  he  incorporates  it  in  his  behavior  with  the  wheelbarrow. 

A conceptual  construct  has  been  required  to  account  for  the  everyday  knowledge 
that  people  adapt  the  literal  content  of  their  experiences  (i.e.,  what  they 
have  learned)  to  variations  in  problem  situations  not  literally  equivalent  to 
previously  experienced  patterns.  The  notion  of  implicit  task  formats  seems  to 
help  account  for  these  adaptations.  The  notion  also  accounts  for  some  of  the 
phenomena  of  improvement  in  task  skills  that  are  recognized  on  a subjective 
basis  but  not  readily  fitted  into  traditional  cognitive  schema. 

In  psychological  terms,  one  should  not  think  of  a task  format  as  a mechanism, 
or  produced  by  a specialized  "format  generator"  in  the  nervous  system.  Rather 
it  should  be  regarded  as  a phenomenon  having,  like  any  other  product  of  learning 
variety  in  kind  and  in  effect.  It  is  to  be  hoped  that  the  label  "format"  is 
not  taken  too  literally  or  mechanistically.  It  is  a logical  construct,  like 
the  constructs  of  attitude  and  motivation,  which  are  tested  not  for  truth  as 
perceivable  entities,  but  for  utility  in  generating  hypotheses  about  practical 
as  well  as  theoretical  matters. 

In  the  practical  community  where  objects  and  procedures  are  designed  for  human 
use,  and  where  training  is  designed  for  programing  these  procedures  into  human 
operators,  the  notion  of  task  standardization  may  acquire  a broader  base.  Some 
designers  are  guided  by  the  philosophy  of  giving  the  operator  a completely  un- 
structured work  environment  with  the  intent  of  giving  him  freedom  and  flexi- 
bility. But  with  repeated  practice,  the  operator  will  develop  spontaneously  a 
structure  for  coping  with  constants  in  variable  situations  in  a constant  way. 

The  risks  in  freedom  are  slowness  in  reaching  criterion  levels  of  task  effi- 
ciency and  the  probability  that  a spontaneously  developed  task  format  may 
usually  be  poorer  than  one  that  has  been  deliberately  designed  with  the  full 
range  of  task  purpose,  task  environments  and  the  properties  of  human  behavior 
in  mind.  On  the  other  hand,  a few  gifted  individuals  may  spontaneously  develop 
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implicit  formats  of  their  own  which,  at  least  for  them,  are  superior  to  their 
use  of  explicit  formats  designed  by  others.  The  risk  in  designing  an  explicit, 
external  format  for  a task  is  that  the  structure  may  be  inappropriate,  or  that 
it  will  unduly  tyrannize  and  stereotype  a task  which  should  be  performed  with 
judgment  and  flexibility.  But  these  are  the  well-known  tradeoffs  that  apply 
to  any  form  of  standardization  whether  in  tasks  or  in  automobile  mufflers. 

Only  to  the  timid  or  uninformed  need  a tradeoff  be  a standoff.  Any  finite 
solution  should  be  aimed  at  a statistical  expression  of  success,  not  an  abso- 
lute one. 

In  any  event,  the  clear  recognition  that  human  operators  can  and  do  generate 
behavior  that  abstracts  and  copes  not  only  with  data,  but  data  variables  in 
procedural  and  conceptual  tasks,  as  well  as  in  tracking  tasks  such  as  aiming 
and  driving  a vehicle,  offers  design  and  training  concepts  that  can  facilitate 
order  in  human  task  learning  and  performance  with  less  than  a straitjacket. 
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GOAL  PROJECTION 


A specification  or  characterization  of  a desired  or 
satisfying  state  of  affairs  which  a system  seeks  to 
acquire  or  maintain  as  shown  by  the  alternatives  it 
selects  (or  avoids)  in  its  choice-making  behavior, 
and  towards  which  it  is  energized  in  its  drive  levels. 

The  goal  projection  or  image  is  the  reference  information  with  which  a system 
compares  a present  or  planned  state  of  affairs.  If  on  comparison  a difference 
is  found,  behavior  is  modified  accordingly  in  direction  and/or  magnitude. 

In  an  automatic  system,  goal  image  or  purpose  is  implicit  in  the  cybernetic 
control  mechanisms.  In  human  systems,  or  systems  that  include  a human  opera- 
tor, purpose  and  goal  state  concepts  may  be  both  implicit  and  explicit,  and 
may  be  partly  independent  of  control  mechanisms. 

An  inanimate  system  may  seek  an  equilibrium  level  and  direction  of  activity 
that  comprises  its  operating  program  with  changing  state  of  the  environment. 

An  animate  system  may  have  a striving  component  for  moving  by  means  of  task 
actions  from  a given  status  to  an  approximation  of  a reference  goal  state. 

The  internal  "reference"  for  the  human  operator  is  a goal  image.  Thus,  a 
goal  image  serves  as  the  embodiment  of  a criterion  for  terminating  a task  or 
segment  of  work  or  mission,  and  terminating  it  with  an  experience  of  some 
degree  of  success  or  failure. 

Different  goal  images  may  apply  to  different  levels  of  activity;  e.g.,  task, 
duty  segment,  work  cycle,  mission. 

Examples 

• The  recruit  with  a clear  picture  of  what  a clean  rifle  should  look 
like,  who  continues  cleaning  it  until,  from  all  suitable  points  of 
regard,  its  appearance  meets  this  criterion. 

• The  workman  who  fabricates  a part  to  within  his  idea  of  what  the 
inspection  tolerances  should  be. 

• The  mechanic  who  services  an  engine  until  it  looks,  sounds,  and 
operates  well  according  to  all  its  performance  variables  as  he 
knows  them. 

• The  pilot  flying  "straight  and  level"  on  instruments  in  a straight 
and  level  maneuver. 

Functions  of  Goal  Image 

The  goal  image  serves  as  both  a steering  and  a power  reference  for  moving  from 
a present  or  projected  state  of  affairs  into  a projected  route  of  action.  Goal 
Information  is  operationally  meaningful  insofar  as  the  behavior  of  the  system 
Is  not  completely  and  rigidly  programmed  or  procedural i zed;  i.e.,  where  the 
system  has  options  for  activity  in  kind  and  amount. 


A-17 


TAEG  REPORT  NO.  12-3 


Goal  Projection 

Where  the  system  operates  on  a performance  criterion  rather  than  on  a fixed- 
cycle  time  criterion,  the  goal  data  indicate  a point  of  work  completion. 

Thus,  the  goal  image  tends  to  determine  the  subjective  condition  of  "closure" 
and  reset. 

Forms  of  Goal  Image 


• With  some  kinds  of  subject  matter,  the  image  may  be  a mental  pictorial 
picture  of  an  idealized  or  satisfactory  terminal  state.  The  image  may 
be  clear  or  vague;  and  it  may  be  clearer  in  terms  of  some  variables 
than  others. 

• The  goal  Image  may  consist  of  criterion  or  dependent  variables  within 
some  wanted  ranges  of  values. 

• The  goal  picture  may  consist  of  indications  of  "progress"  from  a 
present  position  on  one  or  a group  of  criterion  variables,  rather 
than  terminal  values. 

Behavior  Tendencies 


Disregard  of  goal  image.  The  student  and  novice  operator  often  try  to  start 
training  by  learning  procedures  and  manipulations  rather  than  goals  and  goal 
variables.  In  many  cases,  the  instructor  must  insist  on  the  student  learning 
to  distinguish  good  from  poor  goal  models  before  allowing  the  trainee  to  get 
into  the  mechanics  of  the  task. 

Deteriorating  goal  image.  Goal  images  may  become  tentative  and  changeable. 

Change  is  often  in  the  direction  of  deterioration.  This  decay  may  occur  in 
the  course  of  task  and  job  learning  when  the  effort  and  expertise  demanded  for 
a high  level  goal  confront  a low  energy  budget  and  low  student  ability.  Social 
influences  are  well-known  modifiers  of  goal  levels  or  levels  of  aspiration. 

The  goal  image  may  vary  markedly  within  and  between  work  cycles,  under  the 
influence  of  physiological  states,  stress,  work  context,  and  other  factors. 

These  changes  may  be  adaptive  and  functional  in  some  cases.  Goals  that  are 
"spontaneously"  degraded  are  not  likely  to  be  the  best  operating  adaptations, 
however,  so  that  training  may  be  necessary  to  guide  the  operator  in  priorities. 

Disappearance  of  goal  image.  Routinization  tends  to  blunt  the  clarity  and 
dominance  of  the  goal  image,  so  that  it  may  disappear  into  automatic  pro- 
cedural behavior.  This  may  be  adaptive  in  the  sense  of  requiring  reduced  in- 
formation processing  activity  if  the  automatic  performance  is  at  least  accept- 
able and  the  environment  is  higfily  stable  from  one  work  cycle  to  another. 

Such  conditions  tend  to  be  rare. 

Training  and  Performance  Strategies 

The  strategies  set  forth  below  reflect  not  only  the  operational  factors  involved, 
but  also  some  philosophical  and  psychological  biases  of  the  author.  One  is 
that  if  the  operator  does  not  continuously  strive  to  do  a "better"  job  — 
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Goal  Projection 

according  to  his  own  lights,  as  well  as  those  of  others  --  his  performance 
will  tend  to  degrade,  and  he  will  oscillate  just  above  and  below  externally 
acceptable  levels  of  performance  and,  in  a clutch  situation,  he  will  fail. 
Another  bias  is  the  belief  that  a job  or  task  can  remain  interesting  after  . 
acceptable  mastery  has  been  achieved  only  if  the  level  of  aspiration  for  the 
performance  of  the  task  always  goes  somewhat  higher  than  the  actual  achieve- 
ment level.  Even  intrinsically  dull  tasks  can  remain  bearable  if  the  opera- 
tor continues  to  try  to  beat  his  own  record.  Obviously,  the  operator's  work 
environment  should  not  penalize  the  achievement  of  higher  levels  of  perfor- 
mance; at  worst,  the  work  environment  should  be  indifferent  to  such  achieve- 
ment. 

Clear  goal  variables.  Training  should  emphasize  (and  test)  the  student's  early 
acquisition  of  goal  variables.  Where  possible,  emphasis  should  be  placed  on 
concrete  pictures  of  good  goal  states  that  embody  the  desirable  values  among 
the  entire  pattern  of  goal  variables  associated  with  the  task.  The  goal 
images  that  are  presented  should  be  realistic,  not  idealized  ones  that  can  be 
achieved  only  under  idealized  conditions  that  never  occur.  If  goal  images 
presented  are  not  realistic  — realistic  in  terms  of  a skilled  operator  in 
representative  conditions  — the  student  will  become  disillusioned  and 
cynical  of  the  instruction.  The  student  should  be  taught  to  identify  the 
variables  that  comprise  a goal  state. 

Tolerance  ranges.  The  student  should  learn  the  range  of  values  that  would  be 
operationally  acceptable  on  each  criterion  variable.  If  this  is  done,  the 
student  should  be  able  to  distinguish  values  on  each  variable  that  are  barely 
acceptable  and  those  that  are  unacceptable;  he  should  also  be  able  to  dis- 
tinguish limits  in  terms  of  the  composite  goal  picture  as  well.  If  there  are 
operational  tradeoffs  between  goal  variables  (e.g.,  speed  versus  accuracy), 
tne  acceptable  range  within  which  these  tradeoffs  can  be  made  should  be 
clarified. 

Reliably  clear  pictures  of  goal  states,  goal  variables  and  their  tolerance 
ranges  should  precede  training  on  the  performance  of  the  work.  This  is  sound 
training  strategy  because  it  enables  the  student  to  serve  largely  as  his  own 
feedback  mechanism,  a psychologically  good  arrangement  in  that  it  centers 
initiative  in  the  student  rather  than  leaving  it  entirely  in  the  instructional 
mechanism.  The  strategy  also  induces  achievement  motivation  in  the  student 
who  may  maintain  it  after  he  terminates  formal  instruction.  (Contrast  this 
with  the  student  who  continues  improving  only  under  the  direct  guidance  of  an 
instructor.)  Application  of  this  strategy  makes  learning  purposeful  — in 
terms  of  objectives  --  rather  than  mechanical. 

Priorities  and  contexts.  Goal  aspirations  may  have  to  be  compromised  by  opera- 
tional  exigencies.  TRe  operator  should,  therefore,  know  which  criterion 
variables  have  the  highest  priority  for  sustained  integrity  and  which  may, 
when  necessary,  be  degraded  with  least  operational  penalty,  for  given  kinds 
of  missions.  Such  knowledge  enables  one  kind  of  rational  judgment  to  be 
exercised.  It  may  also  enable  a mission  to  be  completed  which  otherwise 
would  remain  undone;  if  there  seems  to  be  no  choice  for  the  operator  to  go 
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ahead  even  in  a degraded  mode,  he  may  flounder  or  cease  to  do  anything  at  all. 

It  is  better,  for  example,  to  demolish  a car  than  to  kill  or  maim  a pedestrian  - 
a choice  which  apparently  some  motorists  have  not  been  trained  to  make 
correctly. 

A concept  of  operational  (and  ethical)  priorities  among  a collection  of  goal 
criteria  is  essential  to  the  higher  levels  of  work  performance.  In  fact, 
strategic  "rules"  for  selecting  goal  priorities  make  a large  number  of  the 
strategies  described  in  this  handbook  as  procedural,  seem  almost  trivial  by 
contrast.  In  simple  terms,  the  issue  is  what  is  more  important,  and  what  is 
less  important  to  achieve  in  a task  or  mission.  By  being  relatively  specific 
on  such  matters,  the  operator  himself  becomes  an  active  strategic  element  in 
the  enterprise  by  being  able  to  maintain  a control  initiative.  In  actual 
practice,  how  often  does  any  worker  get  consistent  instruction  in  goal 
priori  ties? 

Rational  control  over  impulse.  To  the  extent  that  the  operator  has  a clear 
picture  of  goal  states,  and  is  guided  and  driven  by  them,  he  has  a basis  for 
rational  control  over  emotional  impulse  and  the  tendency  for  expedience  at 
the  moment  --  in  other  words,  he  has  a basis  for  "localized  optimization". 

When  the  operator  is  under  strong  pressure  from  an  internal  or  external  source, 
any  device  for  the  control  of  emotional  impulse,  and/or  proper  direction  of 
impulse  as  a source  of  energy,  will  have  operational  payoff.  Obviously,  such 
devices  have  the  highest  payoff  frequency  where  the  stresses  are  greatest  and 
most  frequent.  (Note  that  even  boredom  is  a stress!)  Indoctrination  and  re- 
furbishing of  the  subjective  goal  image,  therefore,  has  a continuing  stra- 
tegic value  for  operator  performance. 

Comment 


These  "strategic  principles"  for  training  and  performance  are  indeed  self- 
evident  and  are  probably  known  in  one  form  or  another  by  the  layman.  It  is 
unfortunate,  however,  that  such  knowledge  seems  so  rarely  to  be  put  into 
systematic  practice,  even  by  specialists  in  the  training  and  educational 
arts  and  by  managers. 
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SCAN  - DETECT 
Is  something  there? 

Procedures  and  mechanisms  for  sensing  the  presence  of 
a cue  or  condition  requiring  that  some  form  of  action 
be  taken  by  the  system.  The  action  may  consist  of 
getting  more  information  about  the  condition  detected. 

Detection  requires  discrimination  of  an  action-stimulating  cue  from  some 
background.  What  is  detected  may  be  occurring  in  the  environment  or  within 
the  system  itself. 

Detection  is  ordinarily  a positive,  consumnatory  response  to  the  activity  of 
scanning.  "Scan"  and  "detect"  should,  therefore,  be  linked  activities  and 
they  are  treated  as  such  here.  Since  effective  detection  Is  largely  a 
function  of  effective  scanning,  major  parts  of  the  discussion  in  this  section 
deal  with  scanning  conditions  and  strategies.  Scanning  is  defined  as  the 
moving  exposure  of  a sensing  device  to  a field  of  possible  signals.  The 
pattern  of  exposure  may  be  systematic  or  random,  periodic  or  aperiodic. 

The  terms  scan,  detect,  search,  identify,  and  interpret  tend  to  overlap 
functionally;  distinctions  among  them  are  somewhat  artificial.  Subtle  dif- 
ferences between  search  and  scanning/detection,  as  well  as  between  detection 
and  identification,  are  suggested  by  the  following  example.  The  operator  may 
be  looking  for  a needle  in~a  haystack,  but  he  can  Identify  the  needle  when  ho 
sees  it,  even  though  he  engages  in  scanning  and  detects  the  glint  of  the  needle 
before  he  identifies  it  as  the  object  he  seeks.  In  performing  a search  func- 
tion, the  operator  has  a particular  objective  in  mind.  Detection  is  the 
sensing  of  an  irregularity;  in  a stimulus  context  ordinarily,  detection  calls 
for  attention  which  leads  to  getting  more  information  that  enables  identifica- 
tion. An  automobile  driver,  for  instance,  may  detect  an  Irregularity  in  the 
"texture"  of  the  roadway  ahead  and  begin  an  evasive  maneuver  before  he  has 
identified  the  irregularity  as  a pothole,  an  old  tire  tread,  or  merely  a 
shadow. 

Examples 


Scanning  and  detection  occur  in  a large  variety  of  contexts. 

• The  indicator  in  an  enclosed  environment  such  as  a cockpit. 

• An  open  field  in  a natural  environment,  such  as  an  aircraft  in  the 
open  sky. 

• A bounded  field  in  an  artificial  representation  of  an  environment, 
such  as  a radar  scope. 

• An  open  field  with  a complex  ground  in  which  a figure  is  to  be 
detected,  such  as  an  enemy  sniper  concealed  in  part  by  foliage 
and  other  terrain  features. 
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• A relatively  closed  field  in  which  some  property  or  attribute  of 
one  or  more  objects  should  be  discriminated;  e.g.,  a blackened 
wire  in  an  electronic  assembly. 

• A collection  of  symbols;  e.g.,  text  which  is  being  proofed  for 
errors,  a wiring  diagram  which  is  scanned  for  an  uncompleted 
connection. 

• In  urban  traffic,  a faint  whining  sound  that  grows  louder  and  is 
identified  as  a police  or  ambulance  siren. 

• A slight  change  of  odor  in  a house  which  is  identified  (once  atten- 
tion is  focused  upon  it)  as  gas  from  a leaking  stove,  or  as  a 
burning  roast. 

/ 

t An  open  bo<fy  of  water  in  which  buoys  must  be  detected  that  indicate 
shoals,  hazards,  and  the  channel. 

Operational  Variables  and  Principles 

• The  lag  or  cycle  time  of  the  scan  and  detect  function/device  must 

be  less  than  the  cycle  time  required  for  the  stimulus  to  be  detected. 

• The  greater  the  contrast  between  the  stimulus  to  be  detected  and  its 
background,  the  greater  the  reliability  of  detection. 

• Some  scan  patterns  and  frequencies  are  better  than  others  for 
detecting  given  kinds  of  signal  patterns. 

• What  the  human  operator  detects  is  related  to  "set"  or  pre-established 
response  tendencies.  More  simply,  we  tend  to  notice  what  we  expect  to 
see,  what  we  are  looking  for,  or  what  we  are  attending  to. 

Discussion 


In  scanning,  both  the  motivational  stimulus  and  the  ongoing  stimulation  for 
what  to  do  next  are  of  internal  origin  to  the  operator,  and  are  sustained  by 
internal  stimulation.  During  much  scanning  time,  there  is  no  response  from 
the  environment  that  serves  as  a cue  for  specific  adjustmental  action.  Unless 
the  operator  has  been  aroused  by  an  emergency  state  — such  as  pain,  hunger, 
fear,  an  alert  for  some  incipient  threat  --  the  human  mechanism  tends  to 
avoid  scanning.  Consequently,  we  should  expect  powerful  behavioral  tendencies 
resulting  in  perfunctory  scanning.  During  periods  that  the  operator  interprets 
as  "steady  state"  conditions  between  himself  and  the  environment,  the  scan 
function  will  tend  to  be  aborted.  Strong  motivations  to  reach  a conceptualized 
or  perceived  goal  reduce  the  quantitative  and  qualitative  scope  of  scanning. 

These  phenomena  are  normal  selective  conditions  of  human  attention  which  are 
generally  adaptive,  but  may  at  times  be  maladaptive. 
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Scanning,  as  a form  of  information  processing  that  must  be  supported  almost 
entirely  by  internal  activities,  is  perhaps  more  vulnerable  to  degradation 
and  poor  performance  than  any  other. dimension  In  task  requirements.  In 
scanning,  the  behavioral  principle  of  progressive  minimization  of  Information 
handled  per  activity  has  the  fewest  off-setting  factors  in  the  task  environ- 
ment on  which  we  can  depend  for  rearousal  of  the  degraded  function. 

Behavior  Tendencies 


The  following  tendencies  occur  with  extended  experience  over  a number  of  task 
cycles,  and  also  with  experience  within  a given  task  cycle.  In  many  respects, 
the  tendencies  are  adaptive  — generally  leading  to  reduced  processing  activity 
by  the  operator  during  task  performance.  In  other  respects,  the  behavior 
tendencies  cited  may  be  maladaptive,  tending  to  reduce  or  delay  the  perception 
of  potential  crisis  conditions. 

Sampling  becomes  standardized.  With  extended  experience,  the  operator  tends  to 
standardize:  what  he  pays  attention  to  and  when;  the  sequencing  of  what  is 
scanned;  and  the  classes  of  cue  which  are  sampled  during  scanning  activity. 
Often,  the  operator  standardizes  what  is  not  looked  for  — and  the  cue  may 
escape  detection,  regardless  of  its  magnitude.  Cues  which  initiate  scan 
maneuvers  may  also  become  standardized. 

Many  scanning  sequences,  which  are  deliberately  invoked  early  in  task  learning, 
become  habitual  and  virtually  involuntary  with  repeated  experience.  This 
tendency  is  adaptive  if.  scanning  habits  continue  to  be  adequate  to  the  work 
conditions,  and  if  alerting  signals  are  detected  as  such.  Since  scan  and 
detect  must  be  a sampling  activity,  it  is  not  likely  to  be  successful  in  each 
and  every  environment.  Failure  probabilities  in  changed  conditions  and  envi- 
ronments, and  the  cost  of  failure  consequences,  may  rise  to  unacceptable  opera- 
tional levels.  In  such  a situation,  "redesign"  of  the  scan  and  detect  function 
becomes  essential.  Scanning  appropriate  to  one  class  of  situation  may  be  quite 
inadequate  in  other  types  of  situations.  For  example,  the  kind  of  situational 
sampling  that  is  appropriate  to  high  speed  turnpike  driving  is  mostly  inadequate 
for  driving  in  dense  urban  traffic,  and  vice  versa. 

Sampling  scope  and  range  become  standardized.  Habit  tends  to  establish  not 
only  what  is  sampled,  but  also  the  range  from  which  cues  are  sampled.  Some 
drivers  seem  to  fix  attention  a standard  distance  ahead,  and  only  upon  their 
intended  path;  some  sample  only  from  the  speedometer,  and  generally  ignore  the 
other  instruments  (fuel  gauge,  tachometer,  etc.).  Even  though  early  training 
may  Induce  more  or  less  appropriate  scanning  behavior,  scanning  tends  to 
degenerate  and  unless  scanning  motivation  and  nabits  are  reinforced,  the 
function  remains  inadequate.  Scanning  habits  (determining  what  is  sampled  and 
the  scope  of  sampling  as  well)  become  firmly  entrenched  and  inadequacies  only 
rarely  are  subject  to  adverse  feedback;  appropriate  scanning  habits  may  quickly 
degenerate  even  after  a period  of  determined  improvement  by  the  operator. 


In  certain  operations,  scanning  habits  may  be  measured  and  corrected.  A look- 
out on  a ship  must  continually  scan  a prescribed  sector  and  report  all  objects 
sighted.  If  the  officer  on  the  bridge  sights  an  object  or  vessel  that  the 
lookout  fails  to  report,  corrective  action  is  taken  to  correct/ improve  scanning 
habits. 

\ 
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Progressive  standardization  of  figure-ground  sampling.  An  experienced  jungle 
fighter  who  has  learned  to  scan  the  brush  for  evidence  of  an  enemy  may,  there- 
by, become  vulnerable  to  enemies  in  towns  and  villages  where  a different  pattern 
of  figure-ground  relations  must  be  sampled  in  order  to  detect  danger.  The 
motorist  who  has  driven  mainly  on  well -paved  superhighways,  fails  to  scan  for 
potholes  in  urban  areas  on  city  streets;  he  would  reliably  detect  an  object 
the  size  of  a pothole  if  it  were  on  the  open  highway,  but  not  in  the  city. 

This  tendency  differs  from  the  preceding  one  only  in  emphasizing  that  humans 
acquire  sampling  habits  specific  to  given  environments  of  figure-ground  relation- 
ships and  variables. 

Reduced  response  to  partial  cues.  Detection  may  be  the  summation  of  a number  of 
cue  samples  obtained  during  a scanning  sequence.  No  one  of  the  cues  in  itself 
may  be  adequate  to  signal  the  need  for  overt  action,  or  even  to  justify  further 
inquiry.  A bored  sentry,  for  example,  may  hear  the  sound  of  a crushed  twig  and 
become  alert  for  a few  moments,  but  return  to  a demi-doze.  Several  minutes 
later,  the  swish  of  dry  leaves  and  the  cessation  of  bird  chirpings  may  arouse 
him  again.  None  of  these  cues  by  itself  is  likely  to  be  sufficient  to  justify 
arousal  and  investigation  or  alarm.  Together,  they  should  indicate' the  approach 
of  a body.  Habitual  absence  of  perceived  threat  reduces  the  tendency  to  link 
threshold  cues  that,  put  together  in  short-term  memory,  would  sum  into  an  aware- 
ness of  a condition  requiring  full  attention  and  action. 

Projection  of  expectations.  Under  conditions  of  strong  expectations,  spurious 
or  random  cues  in  the  environment  may  trigger  incorrect  detections  and  identi- 
fications. The  more  complex  the  ground  and  the  greater  the  difficulty  in 
differentiating  figure  from  ground,  and  the  stronger  the  anticipation,  the 
greater  the  liability  of  spurious  detections.  This  tendency  is  a case  of  the 
interpretive  process  interacting  with  the  detection  process.  Thus,  under  strong 
anticipation  of  an  energy  attack,  many  sounds  and  sights  will  be  detected  as 
cues  that  would  otherwise  fail  to  be  detected.  Circumstances  of  this  sort  can 
also  lead  to  over-response.  When  a military  unit  is  engaged  in  an  assault  on 
an  enemy  position,  a tense  soldier  may  detect  movement  out  of  the  corner  of 
his  eye  and  turn  and  fire  in  that  direction.  Only  instantaneous  positive 
identification  prevents  him  from  shooting  a buddy.  Similar  conditions  occur 
in  driving.  When  a driver  anticipates  an  engine  breakdown,  he  will  detect  as 
unusual,  sounds  and  car  behaviors  which  are  quite  normal  for  the  car  and  to 
which  he  would  pay  no  attention  when  he  is  not  apprehensive. 

Scanning  and  rate  of  change.  This  tendency  is  an  important  subset  of  stan- 
dardized sampling.  Scanning  patterns  and  sampling  rates  tend  to  become  asso- 
ciated with  some  prevailing  rate  of  change.  Scanning  rates  appropriate  for 
driving  at  forty  miles  per  hour  are  inadequate  at  a speed  of  seventy  or  more 
miles  per  hour.  The  driver  who  has  spent  several  hours  on  a winding  two-lane 
road  at  a relatively  slow  speed  is  at  a disadvantage  when  suddenly  accelerating 
to  limited  access  freeway  speeds.  He  is,  in  fact.  In  danger  on  his  approach 
to  the  freeway,  for  this  reason  and  others.  The  same  difficulty  appears  in 
the  shift  in  scanning  activity  required  in  the  area  of  small  local  airfield 
compared  to  that  necessary  in  approaching  a large  metropolitan  area  airport 
which  services  jet  traffic. 
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Tunnel  perception.  In  tunnel  perception,  the  field  of  scan  is  markedly  reduced, 
and  tends  to  be  limited  to  the  intended  goal  path.  Tunnel  perception  is  an 
extension  of  the  normal  conditions  of  attention  which  restrict  what  one  attends 
to  in  terms  of  a goal,  or  to  the  execution  of  a highly  practiced  chain  of 
responses.  It  is  an  extreme  form  of  preoccupation.  A variety  of  conditions 
tend  to  create  tunnel  perception:  fatigue;  anxiety;  extremely  high  motivation 

toward  a goal;  preoccupation  with  other  than  the  task  at  hand;  extreme  routiniza 
tion;  and  repetition  of  a series  of  standard  stimulus-response  sequences. 
Physiological  depressants  (including  alcohol),  no  doubt,  also  affect  tunnel 
perception. 

Decreases  in  effective  scanning  and  detection  behavior  may  be  one  of  the  most 
sensitive  means  of  identifying  the  impairment  of  an  operator  or  of  his  skills. 
(Impairment  in  short-term  memory  capacity  is  another.)  Because  scan  and 
detect  functions  are  generally  tested  by  relatively  rare  events,  this  impair- 
ment in  scan-detect  behavior  may  not  be  observable  in  the  operations  of  the 
individual  operator,  but  may  appear  only  in  statistical  ensembles  of  task 
samples. 

Performance  and  Training  Strategies 

The  way  to  train  an  operator  for  alertness  and  assure  that  alertness  will 
persist  indefinitely  after  formal  training  is  not  clear.  In  Infantry  training, 
a soldier  with  his  rifle  loaded  walks  down  a trail  in  a forest.  He  must 
instantaneously  respond  to  targets  that  appear  suddenly.  He  must  identify 
these  as  friend  or  enemy  and  if  enemy,  fire  immediately. 

In  nature,  animals  deficient  in  scan  and  detect  mechanisms  (commensurate  with 
the  speed  and  power  of  their  fight  or  flight  capabilities)  tend  to  have  a brief 
life.  In  human  societies,  such  selective  procedures  are  less  ruthless.  More 
compensatory  mechanisms  tend  to  keep  the  occasional  bobble  that  is  due  to  in- 
attention from  becoming  a catastrophe. 

It  is  clear  that  the  learning  of  effective  scanning  behavior  requires  active 
and  continuing  cooperation  of  the  student  and  continuing  initiatives  of  the 
student/operator.  He  must  not  only  acquire  effective  scanning  and  detection 
habits,  but  also  recognize  various  conditions  that  lead  to  deterioration  of 
these  habits  so  that  he  can  compensate  for  them. 

The  following  are  some  training  implications  based  on  the  liabilities  cited 
above. 

Cognitive  models  of  threat.  In  operating  any  vehicle,  there  is  a zone  of 
threat  involved.  Sate  driving  consists,  in  part,  of  keeping  as  large  a peri- 
meter of  safety  against  potential  threat  as  practicable.  This  perimeter  is 
not  symmetrical;  it  is  greater  to  the  front  than  on  the  other  three  sides. 

The  higher  the  speed,  the  greater  the  boundary  of  the  safety  zone.  If  one 
must  stop  quickly,  the  rear  presents  a hazard.  The  skilled  operator,  perhaps 
by  sampling  every  few  seconds,  is  aware  of  the  location  of  other  cars  including 
oncoming  traffic.  In  case  there  is  a need  for  a rapid  emergency  response,  his 
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awareness  of  the  location  of  "competitors  for  his  space"  provides  him  with 
the  greatest  number  of  options  for  maneuver,  and  the  information  for  choosing 
instantaneously.  By  continuous  sampling  from  the  entire  region  of  his  task 
operations,  he  is  aware  not  only  of  the  potential  threats,  but  of  his  actual 
response  options.  This  might  well  be  thought  of  as  an  operational  model  of 
the  "lifespace"  of  his  task  environment.  The  greater  the  driver's  skill,  of 
course,  the  better  he  can  cope  with  constricted  lifespace  --  down  to  some 
limit. 

Highly  versatile,  programmable  simulators  may  be  developed  which  can  generate 
realistic  situations  that  enable  the  operator  to  achieve  effective  sampling 
rates  for  various  configurations  in  the  environment.  The  relative  frequency 
of  hazards  posed  by  the  simulator  would  have  to  be  far  greater  than  what  the 
operator  can  expect  in  real  life.  But  he  would  be  learning  to  cope  with 
non- representative  situations  --  to  anticipate  through  scanning  and  detection 
and  perhaps  to  avoid  threatening/hazardous  situations. 

A cognitive  model  of  threat  is  a representation  of  the  dynamic  factors  in  a 
task  that  need  to  be  sampled  at  some  frequency  and  range  in  order  for 
probable  survival.  Sampling  that  enables  prediction  of  an  event(s)  generates 
the  largest  range  of  options  for  coping  with  the  event  if  and  when  it  does 
materialize. 

A model  of  opportunities,  the  converse  of  a model  of  threat,  is  a "map"  of 
the  factors  to  be  sampled  by  the  operator  in  order  to  increase  chances  for 
a successful  outcome.  The  map  must  include  representation  of  events  in 
real  task  time  to  enable  the  student  to  acquire  habits  of  scanning  and 
detection,  rather  than  merely  ideas  about  them. 

Scan  redundancy.  If  time  permits,  before  committing  himself  to  a response 
in  a dangerous/serious  situation,  the  operator  should  scan  his  environment 
several  times  for  obstacles,  or  for  competitors  of  the  path  he  is  choosing, 
and  for  a potential  escape  route  if  his  maneuver  fails.  Scan  redundancy 
permits  the  operator  to  evaluate  the  relative  motion  of  his  and  other  vehicles. 
The  scan  redundancy  performance  strategy  is  especially  significant  when  moving 
from  one  field  or  rate  of  action  to  another.  Critical  information,  missed 
in  one  scan  (sample),  may  be  caught  or  sumrnated  in  a second  or  third  sample. 
"Look  before  you  leap,"  should  perhaps  be  amplified:  "Look  twice  before  you 

take  a long  leap  over  a deep  chasm." 

Scan  redundancy  may  be  especially  important  if  the  operator  is  in  an  impaired 
state;  e.g.,  fatigued,  anxious,  strongly  goal-driven,  affected  by  severe 
stress,  or  emerging  from  preoccupation  with  a particular  matter. 

Counteracting  blind  habit.  Since  routine  habitual  activities  tend  to  elicit 
tunnel  perception,  the  operator  must  generate  and  maintain  initiatives  for 
scanning;  he  must  scan  the  environmental  field  for  liabilities  which  may 
not  have  occurred  in  his  past  experience.  Occasional  refresher  training 
which  includes  programmed  simulated  hazards  may  help.  Such  training  is, 
however,  likely  to  be  both  brief  and  somewhat  artificial,  hence  of  limited 
effectiveness.  There  seems  to  be  no  external  operation  that  would  serve 
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as  an  adequate  alternative  to  (either  replacing  or  supplementing)  operator 
initiatives.  The  operator  himself  must  periodically  check  his  own  scanning 
skill,  and  deliberately  modify  and  improve  it  in  order  to  prevent  deteriora- 
tion. 

An  operator  may  go  through  the  motions  of  habitual  scanning  and  see  nothing; 
if  so,  the  detection  process  is  not  at  work.  The  operator  needs  occasionally 
to  test  himself  on  this  score.  Noting  that  he  has  just  looked  at  the  rear 
view  mirror,  for  instance,  he  may  ask  himself  "What  is  behind  me?"  If  a 
moment  after  scanning  he  cannot  recall  what  he  saw  in  the  mirror,  he  is  not 
processing  effectively.  A friend  of  mine  was  overheard  telling  an  associate 
that  he  had  driven  to  work  that  morning,  as  he  had  for  years,  but  couldn't 
remember  a thing  about  the  entire  trip.  This  was  disconcerting  to  him;  he 
decided  to  revise  his  processing  activities  including  scan  and  detect  be- 
havior while  driving. 

Awareness  of  impairment.  Aware  that  he  is  drowsy,  tired,  or  tense,  the  opera- 
tor  may  compensate  by  deliberately  putting  more  effort  into  scanning  to  avoid 
tunnel  perception.  If  he  is  unable  to  do  so,  he  should  take  the  operational 
precautions  available  to  him.  The  motivation  for  doing  so  may  be  stimulated 
by  information  regarding  the  effects  of  failure  to  respond  to  the  signs  of 
impairment.  This  is  one  basis  for  highway  safety  ads  on  TV  and  in  the  news- 
papers that  graphically  and/or  verbally  depict  consequences  of  drunk  driving, 
not  wearing  seat  belts,  etc. 

Ambiguity  and  stress.  Under  conditions  that  combine  ambiguity  of  environmental 
cues  and  operator  stress,  the  operator  may  become  dogmatic  about  what  he  has 
detected  and  apparently  identified.  The  rational  strategy  under  conditions 
of  stress  and  ambiguity  is  to  consider  a conclusion  as  merely  an  hypothesis 
and  "collect  more  data"  --  by  further  scanning  for  cues  that  support  or 
nullify  the  earlier  conclusion.  This  strategy  demands  a high  level  of 
rationality,  learning  from  a series  of  recent  unpleasant  experiences,  or  a 
combination  of  both.  The  application  of  this  principle  demands  an  unusual 
form  of  professionalism  on  the  operator's  part. 

Peripheral  scanning.  In  some  time-constrained  tasks,  it  may  be  desirable  to 
scan  a wide  perimeter  even  though  little  time  is  available  for  doing  so. 
Instruction  and  exercises  can  be  directed  towards  increasing  the  effective- 
ness of  scanning  with  peripheral  vision.  Football  and  basketball  coaches 
teach  this  with  apparent  success. 

It  is  possible  to  detect  an  object  peripherally  without  being  able  to  identify 
it.  It  is  also  possible  to  detect  an  object  in  central  vision  but  be  unable 
to  identify  it  (due,  for  example,  to  lack  of  time,  urgency,  etc.).  This 
phenomenon  has  been  seen  in  tachistoscopic  experiments.  Under  time  pressure, 
a driver  who  is  unaccustomed  to  the  use  of  a side  mirror  may  glance  rapidly 
over  his  left  shoulder  and  return  his  gaze  to  the  roadway  ahead  in  a fraction 
of  a second;  this  activity  may  be  sufficient  to  detect  a shape  but  insuffi- 
cient to  identify  the  shape.  - Identification  may  not  be  necessary  and  may, 
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in  fact,  be  counterproductive  if  split-second  reaction  is  necessary.  It 
is  unlikely  that  any  novice  to  a task  will  refrain  from  the  temptation  of 
spending  the  time  to  identify  something  which  he  has  detected.  The  extra 
few  tenths  of  a second  he  takes  may  mean  the  loss  of  an  option  for 
successful  evasive  action.  The  advantage  of  a few  tenths  of  a second 
multiplied  by  many  exposures  can  have  a significant  effect  on  accident  and 
other  failure  statistics. 

In  time-critical  circumstances  (and  perhaps  others  as  well),  the  operator 
should  gather  no  more  information  about  the  situation  than  is  necessary 
for  selecting  an  effective  operational  response  or  discarding  an  ineffective 
one. 

Gaos  in  scanning.  Like  other  behavior,  scanning  is  likely  to  have  discon- 
tinuities in  effectiveness,  gaps  between  fixations  --  especially  in  visual 
fields  that  are  noisy  and  without  reliable  references.  A set  of  super- 
imposed grid  lines  is  an  example  of  a reference  for  scanning.  The  scanning 
problem  may  be  that  of  trying  to  detect  crash  survivors  in  a turbulent  body 
of  water  from  an  aircraft.  Two  scanning  strategies  may  be  used  to  ensure 
scan  "interleaving".  One  is  to  deliberately  overlap  segments  of  the  area 
scanned.  This  principle  is  used  in  radar  scanning  for  early  detection  and 
warning  systems.  A second  strategy  is  to  successively  scan  the  same  field 
with  different  scan  patterns,  one  vertical  sweep  and  once  diagonally  across 
the  field.  If  the  field  has  a definite  directional  texture,  three  scan 
patterns  over  the  same  field  are  more  likely  to  be  effective  --  one  with 
the  grain,  one  against  it,  and  one  across  the  grain. 
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What  is  it  and  what  is  its  name? 

Methods  for  characterizing  a message  by  type  or 
source,  used  ordinarily  to  recognize  an  object 
or  entity  and  apply  a label  to  it. 

The  definition  extends  to  a variety  of  contexts  in  which  the  identification 
function  operates.  Identification  requires  a referencing  action  which  pro- 
duces the  name  or  same  equivalent  symbolic  response  to  be  attached  to  the 
sensed  input.  In  human  behavior,  the  content  of  this  symbolic  response 
may  not  always  be  explicit.  One  may  "recognize"  an  individual  even  though 
his  name  is  not  recalled,  nor  the  explicit  situation  in  which  you  met. 

In  automatic  information  processing,  two  sets  of  reference  codes  may  be 
necessary  for  adequate  identification.  One  reference  structure  may  apply 
to  the  universe  outside  the  processing  system  (for  example,  the  name  and 
address  of  the  sender  of  the  message).  The  other  reference  structure  is 
to  the  physical  and/or  functional  location  of  the  message  within  the  system. 
These  two  identity  codes  may  require  a set  of  cross-referencing  rules  or 
codes . 

Discussion  of  the  Behavioral  Process 


The  learning  of  an  "identification"  consists  essentially  of  two  parts: 
learning  to  recognize  a pattern  of  cues  that  constitute  a real  or  conceptual 
entity  (which  may  be  a process  or  condition  as  well  as  a thing,  or  the  class 
to  which  the  thing  belongs);  and  learning  the  name  to  be  attached  to  the 
entity  and  to  associate  the  name  with  the  entity,  the  entity  with  the  name. 

The  name  can  serve  in  short-term  memory  as  a transcript  code  for  the  thing 
or  condition  conceptualized.  This  transcript  or  code  may  take  up  less 
"space"  in  short-term  memory  than  might  be  required  to  hold  the  image  of 
the  entity  in  mind,  especially  where  a list  of  items  must  be  kept  in  mind 
for  procedural  processing. 

The  learning  of  an  identification  combines  the  processes  of  acquired  dis- 
crimination and  acquired  generalization.  Within  the  context  of  a succession 
of  experiences  with  the  entity,  cues  are  sampled  which  become  at  least 
sufficient  for  discriminating  the  entity  from  competing  entities  --  again, 
within  the  experience  pattern  of  the  learner.  A name  or  other  symbolic 
gesture  becomes  associated  with  the  cue  pattern.  With  extensive  repetitions, 
only  structural  elements  in  the  cue  pattern  sufficient  for  the  identification 
(in  the  experienced  contexts)  tend  to  be  used.  This  abstract  of  cues  gives 
rise  to  the  object  constancy  phenomena.  It  leads  to  progressively  fewer 
cues  being  sampled  for  making  the  identification,  down  to  some  limit 
adequate  for  discrimination  of  the  entity.  The  abstracting  process  fosters 
stereotyped  perceptual  response.  The  operator  may  have  difficulty  in 
actually  "seeing  the  object  as  presented  to  him"  in  terms  of  actual  sensory 
input;  he  sees  substantially  what  he  expects  to  see. 
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If  an  individual  is  required  to  identify  individual  members  in  a class  of 
members  with  which  he  is  unfamiliar,  another  process  seems  to  take  place. 
Consider  an  American  visiting  a Chinese  city.  At  first,  most  of  the  faces 
seem  alike;  he  has  difficulty  in  remembering  the  distinction  between  the 
doorman  and  desk  clerk.  In  a few  days  or  weeks,  this  difficulty  tends  to 
disappear.  It  is  as  if  he  developed  a new  perceptual  matrix  for  differ- 
entiating members  within  the  new  object  class,  and  thus  for  identifying 
individual  entities.  It  is  unlikely  that  learning  a new  perceptual  "set" 
or  perceptual  matrix  can  be  facilitated  by  deliberate  cognitive  analysis 
of  the  new  matrix.  (Recall  the  aircraft  identification  procedures 
developed  in  World  War  II  which  were  effective  when  samples  of  the  entire 
aircraft  species  were  displayed  for  recognition  of  the  pattern,  but 
relatively  ineffective  when  taught  by  analytic  methods.) 

Every  work  environment  will  have  several  classes  of  entities,  the  members 
of  which  must  be  identified  by  the  operator.  If,  as  a student,  the  operator 
does  not  learn  the  identifications  of  the  tools,  objects,  or  operations  of 
his  work  environment,  he  is  handicapped  in  learning  work  procedures  and  in 
remembering/ fo 11  owing  instructions  that  deal  with  these  entities.  That  is, 
he  lacks  subjectively  clear  pairings  of  names  and  image  with  which  to 
translate  the  words  in  the  instructions  into  a picture  of  what  is  to  be 
done,  or  to  transalte  the  appearance  of  a work  object  into  a name  which 
can  be  linked  to  a verbalized  instruction. 

Behavioral  Tendencies 


The  following  are,  for  the  most  part,  normal  behavior  tendencies  which  can 
sometimes  be  liabilities  occurring  at  different  levels  of  training  and 
experience. 

Recognition  despite  noise.  The  operator  fails  to  recognize  an  object  pre- 
sented in  the  presence  of  perceptual  noise,  unless  the  correct  conceptual 
matrix  is  presented.  The  perceptual  matrix  is  the  equivalent  of  an 
appropriate  class  set.  Consider  the  case  of  a noisy  radio  channel  over 
which  words  in  English  speech  become  barely  recognizable.  Some  French  words 
are  spoken.  Although  the  listener  may  comprehend  French  nearly  as  well  as 
English,  he  tends  not  to  recognize  the  French.  Once  some  pattern  of  sounds 
becomes  audible  he  may  begin  to  recognize  the  French  words. 

Differentiation  of  cues  within  an  entity.  When  the  Identification  of  an 
object  has  been  highly  overlearned  in  experience,  the  operator  tends  to 
fail  to  differentiate  cues  within  the  identified  entity  and/or  between 
members  in  the  entity  class.  Failure  continues  unless  and  until  a new  per- 
ceptual matrix  for  such  differentiations  is  learned,  or  invoked,  requiring 
that  such  differentiations  be  made.  An  indifference  zone  in  perception 
tends  to  develop  within  which  variations  in  an  object  are  not  noticed.  For 
Instance,  unless  maintaining  a clean  car  is  a matter  of  concern,  one 
normally  does  not  notice  the  day-by-day  variation  in  dirt  that  accumulates. 
Habitual  recognition  leads  to  a reduction  in  sampling  of  presented  cues. 
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Projection  of  expectations.  Identifications  of  members  of  a class  of 
entitles  can  perceptually  degenerate  into  the  sampling  of  minimum  cues 
that  confirm  a perceptual  stereotype.  Failure  to  differentiate  members 
of  the  class  results,  of  course,  in  failure  to  differentiate  experiences 
from  different  members  of  the  class.  Hence,  the  operator  is  unable  to 
profit  from  such  experiences.  In  order  to  reverse  this  state  of  affairs, 
the  operator  must  be  induced  to  pay  attention  to  cues  he  has  learned  to 
ignore.  The  stereotyped  perception  must  be  inhibited.  Changed  stimulus 
sampling  habits  must  be  substituted.  Even  assuming  cooperative  effort  on 
the  operator's  part,  these  changes  can  best  be  brought  about  in  a new 
work  environment  which  will  not  provide  associative  support  for  the  old 
perceptual  habits. 

Attributes  sampled.  After  considerable  experience  on  a task,  identifica- 
tions  by  the  operator  become  limited  to  specific  attributes  of  the  entity 
essential  to  the  task.  If  a clerk's  job  is  to  sort  orders  on  the  basis  of 
the  color  of  paper  on  which  they  are  printed  or  written,  only  that  feature 
of  the  orders  will  be  identified;  he  will  tend  to  ignore  other  attributes 
(e . g . , kind  of  paper,  written  content,  texture,  and  so  on).  Here  too,  if 
the  operator's  task  is  changed  so  that  he  must  consistently  note  and 
identify  an  additional  characteristic  or  two,  task  retraining  virtually 
from  scratch  may  be  required  in  order  to  assure  reliable  performance  -- 
and  there  may  be  occasional  regressions  after  the  new  discrimination  has 
become  psychologically  "automated". 

Identifiers  emphasized.  The  art  of  the  caricaturist  consists  in  perceiving 
and  overemphasizing  the  pattern  of  distinguishing  characteristics  of  people 
and  other  subjects.  The  ease  with  which  the  person  caricaturized  is 
recognized  suggests  a general  tendency  for  selective  emphasis  of  shape 
elements.  This  selective  emphasis  on  distinguishing  characteristics  has  a 
functional  value  in  retrieval  of  image  content  from  memory,  and  suggests 
an  hypothesis  for  training.  Training  for  identification  of  members  of  a 
class  of  objects  should  include  artificial  enhancement  of  distinguishing 
cues  early  in  training,  and  progressive  elimination  of  enhancing  cues  as 
training  proceeds.  Caricature,  in  this  sense,  would  be  useful  only  where 
a large  number  of  class  members  must  be  differentiated  and  identified, 
and  where  it  is  desirable  to  have  the  student  learn  these  identifications 
without  the  benefit  of  working  contexts.  The  policeman  who  is  required  to 
memorize  the  appearance  of  a large  number  of  fugitives  is  a case  in  which 
caricaturization  would  be  useful. 

Identification  without  naming.  It  is  possible  to  identify  a perceived 
entity  without  recalling  its  name.  ("I  recognize  your  face  but  I can't 
remember  your  name.")  If  the  name  is  also  given,  the  operator  may  recognize 
it  as  belonging  to  the  percept.  Work  situations  exist  in  which  it  is 
important  that  name  and  percept  be  reliably  and  rapidly  associated  and 
interchanged.  In  such  work  situations,  practice  of  recognition  that  re- 
quires overt  naming  when  the  object  is  present,  or  the  percept  recalled, 
should  be  required  in  benign  work  circumstances.  There  may  be  large 
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individual  differences  in  ability  to  recall  these  associated  elements; 
some  persons  may  be  more  prone  to  forget  the  name,  others  to  forget  the 
image  --  face  --  that  goes  with  the  name.  One  can  readily  Imagine  the 
difficulties  which  would  occur  in  an  operating  theater  if  a surgeon  could 
not  readily  recall  the  names  of  the  tools  he  needs  to  perform  the 
necessary  activities. 

Identification  by  progressive  subsetting.  Scanning  may  lead  to  the 
detection  and  identification  of  a hand  tool  lying  in  the  grass  beside  a 
workshed.  Approach  and  further  scanning  leads  to  identification  of  the 
hand  tool  as  a screwdriver.  Still  further  scanning  permits  the  tool  to 
be  identified  as  a phillips  head  screwdriver.  It  is  assumed  here  that  as 
the  operator  approaches  the  object,  progressively  more  cues  become  available. 

In  many  cases,  an  object  or  situation  may  be  scanned  and  subsetted  for  the 
primary  purpose  of  determining  whether  the  object  justifies  concern  (that 
is,  indicating  whether  the  object  is  instrumental  to  some  current  functional 
or  potential  purpose)  or  justifies  indifference  and  no  further  attention. 

The  example  cited  is  not  intended  to  imply  that  the  logical  order  from 
more  general  to  more  specific  classification  is  necessarily  characteristic 
of  the  psychological  process.  Rather,  the  sequence  tends  to  be  determined 
by  the  kinds  of  cues  that  are  perceived  and  integrated  into  the  pattern  of 
cues  making  up  the  entity  as  an  object  or  condition  for  further  Interest. 
Gestalt  theorists  have  emphasized  the  integrative  aspects  of  perception  as 
fundamental  and  essential.  For  practical  operational  purposes,  adoption  of 
an  either-or  choice  does  not  seem  necessary. 

Studies  of  pattern  recognition  generally  point  to  the  conclusion  that  if 
entities  (enemy  versus  friendly  aircraft,  for  instance)  must  be  rapidly 
identified  from  minimum  cues  in  any  of  various  orientations,  learning  the 
entire  pattern  enables  identifications  to  be  made  from  fewer  cues  and  with 
more  speed  than  by  analytic  cues.  But  these  results  can  also  stem  from  a 
well-known  principle  --  the  more  closely  conditions  of  the  training  stimulus 
simulate  those  of  the  task  stimulus,  the  greater  the  transfer  of  training. 
(This  may  not,  of  course,  be  the  whole  story.)  In  addition  to  learning  to 
perceive  the  entire  pattern  all  at  once,  analytic  training  has  value  in 
teaching  the  operator  to  test  perceptual  hypotheses  — when  time  and  oppor- 
tunity permit.  A general  pattern  or  shape  may  be  "identified"  as  an  enemy 
aircraft  in  the  sky;  the  identification  may  be  an  hypothesis  that  requires 
some  confirmation.  When  fuselage  cues  become  visible,  analytic  knowledge 
that  enemy  aircraft  have  a characteristic  bend  in  the  fuselage,  whereas 
similar  but  friendly  aircraft  have  a straight  fuselage,  can  enable  positive 
confirmation  of  the  identification.  In  World  War  II,  American  infantrymen 
in  Europe  never  wore  overcoats.  German  infantrymen  did.  The  silhouette 
pattern  at  night  provided  the  basis  for  enemy  identification. 

This  discussion  suggests  that  identification  of  entities  that  are  fully  and 
unambiguously  presented  may  profit  from  a somewhat  different  training 
regimen  than  Identification  of  objects  of  features  under  conditions  of  what 
might  be  called  "stimulus  deprivation",  i.e.,  where  the  object  is  obscured, 
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incompletely  displayed,  or  displayed  at  near- threshold  sensory  values.  In 
the  latter  condition,  recognition  at  any  stage  should  tend  to  be  accepted 
as  an  hypothesis,  and  scanning  should  be  directed  toward  cues  that  confirm 
or  deny  the  hypothesis.  Directed  scanning  would  be  based  on  the  operator's 
ability  to  apply  analytic  operations  to  the  scan  activity  and  to  assemble 
partial  cues  into  a test  of  a recognition  hypothesis.  Our  discussion  in  a 
preceding  paragraph  about  the  use  of  the  emphasized  identifier  would  apply 
here. 

Task  Strategies 


The  following  strategies  assume  that  the  task  demands  reliable  and  correct 
identifications,  that  recognition  habits  tend  toward  low  levels  of  cue 
sampling,  and  that  tests  for  a tentative  identification  are  desirable. 

Cue  sampling.  Taking  into  account  the  risks  and  penalties  in  delayed  identi- 
fi cations,  attempt  to  get  more  than  a minimal  variety  of  cues  from  the  entity 
in  question  before  making  an  identification.  Cues  may  include  the  environ- 
mental or  behavioral  context  of  the  entity.  Adding  more  cues  Increases  the 
probability  of  correct  identification  --  assuming  the  operator  has  an 
adequate  reference  concept  or  percept  of  the  entity.  In  other  words,  ob- 
serve the  entity  with  full  rather  than  perfunctory  attention.  The  cost  in- 
volved may  consist  of  greater  effort  or  concentration  rather  than  extended 
response  time. 

Stress.  Cue  sampling  for  identifications  may  become  especially  poor  under 
any  of  a variety  of  stress  conditions.  If  an  operator  is  aware  of  this 
tendency,  he  may  deliberately  compensate  for  the  liability  by  more  intensive 
checking  for  confirmatory  identification  cues.  If  a patrol  in  enemy 
territory  is  under  acute  stress,  deliberate  effort  needs  to  be  made  to 
differentiate  the  color  of  the  uniform  of  an  approaching  soldier  so  as  not 
to  shoot  a friend.  This  strategy  has  a tradeoff  — not  shooting  a friend 
versus  shooting  before  the  enenjy  gets  close  enough  to  do  damage. 

Forgetting  the  name  or  the  percept.  As  noted  previously,  one  may  forget 
the  name  that  goes  with  the  percept  or  forget  the  percept  that  goes  with  the 
name.  The  recall  strategy  is  to  search  memory  for  associative  context. 

(The  reader  should  refer  to  Recall  Enabling  Information  (Long  Term  Memory) 
strategies. ) 

An  obstacle  or  block  in  recognition  of  an  object,  or  in  recall  of  a name, 
may  be  learned  and  persist  in  future  encounters  with  the  object,  even 
though  its  name  is  learned  again  following  each  experience.  A formal  re- 
conditioning process  may  be  necessary  to  assure  reliable  recall  when  such 
a block  occurs.  (No  Freudian  interpretation  is  necessary.) 

Tests  of  an  identification.  Although  it  is  logically  impossible  to  certify 
absolutely  that  two  entities  are  identical,  it  is  logically  possible  to 
prove  non-identity  of  individuals.  (The  reader  may  recognize  the  same 
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logic  in  the  null  hypothesis  in  statistical  reasoning  --  one  cannot  prove 
the  null  hypothesis  because  one  cannot  logically  prove  from  finite  samples 
"no  difference"  between  populations.)  Tests  of  blood  type  cannot  prove 
that  the  samples  of  blood  were  taken  from  the  same  man;  they  can,  however, 
prove  that  the  samples  were  taken  from  different  men.  Thus,  in  a difficult 
identification,  a single  negative  cue  ma^  be  given  more  weight  in  rejecting 
the  hypothesis  of  an  identification  than  several  positive  cues  may  have  for 
accepting  the  hypothesis  of  identity.  There  are,  of  course,  limitations  in 
the  practical  operational  value  of  the  strategy  of  negative  signs.  Unlike 
blood  type,  many  identifying  cues  can  be  altered.  The  strategy  should  be 
applied  with  an  awareness  of  facts  and  a sense  of  probabilities. 

Key  differentiators.  Some  variables  are  more  important  than  others  in 
establishing  an  identification  or  in  rejecting  a tentative  identification. 

Key  variables  in  the  classes  of  entities  in  the  work  situation  should  be 
learned  and  applied.  Knowledge  of  secondary  variables  may  support  the 
testing  of  a tentative  identification  when  correct  identification  is  important. 

If  the  analytic  process  must  be  used  in  reaching  or  testing  an  identification, 
categorization  strategies  may  be  applicable. 

Training  Implications 

Percept  and  name.  The  learning  and  practice  of  some  jobs  is  heavily  dependent 
on  reliable  memorization  of  a large  number  of  identifications.  Medicine  is  an 
example.  The  student  must  learn  to  recognize  the  appearance  of  the  entity  or 
condition  and  to  distinguish  it  from  the  appearance  of  other  similar  entities. 
He  must  also  learn  the  appropriate  name  or  symbol  for  the  entity  and  differ- 
entiate it  from  other  names  in  the  vocabulary.  He  must  learn  to  think  of  the 
object  when  he  sees  or  hears  the  name,  and  think  of  the  name  when  he  sees  the 
object.  The  operator  may  have  to  recognize  the  object  in  a variety  of  per- 
ceptual and  operational  contexts,  sometimes  in  difficult  perceptual  environ- 
ments. These  constitute  variables  in  effective  learning  for  identifications. 

To  these  variables,  we  must  add  individual  differences  in  the  ability  to 
learn  and  remember  percepts  and  in  the  ability  to  learn  and  remember  names 
and  other  symbolic  labels.  If  the  student  has  a deficiency  in  either  of  these 
abilities,  his  training  should  clearly  have  a different  emphasis  --  assuming 
that  training  is  to  be  fine-tuned  to  a well-defined  set  of  "identifications", 
as  a working  task  vocabulary  preliminary  to  or  concomitant  with  learning  of 
task  procedures.  If  the  conditions  of  training  impose  a heavy  requirement 
for  verbal  learning  as  practice  for  task  performance,  this  requirement  is 
imperative.  (It  is  also  a requirement  for  useful  conceptual  learning  — in 
this  type  of  material,  "identification"  is  the  equivalent  to  "definition  of 
the  concept".) 

Some  individuals  with  good  verbal  memories  may  be  able  to  learn  the  verbal 
content  of  procedures  relatively  quickly  and  may,  therefore,  do  well  on  early 
classroom  tests.  Deficiency  in  the  translation  of  verbal  content  into  per- 
ceptual concepts  permits  individuals  with  poorer  verbal  memories,  but  better 
perceptual  and  perceptual -motor  memories,  to  overtake  those  with  good  verbal 
memories  in  actual  task  performance. 
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If  computer  assisted  instruction  has  a graphic  capability,  it  becomes  an 
excellent  instructional  device  for  adapting  individual  differences  in  rate 
and  kind  of  learning  of  the  identifications  associated  with  a task,  and 
for  achieving  adequate  criteria  of  learning,  at  least  outside  the  opera- 
tional context  of  the  actual  work  itself. 

Enhanced  distinguishabil ity.  The  distinguishabil i ty  of  the  percept  of  the 
object  can  be  enhanced  by  simplification  and  emphasis  on  the  object's 
primary  characteristics;  i.e.,  caricature.  Diagrams  in  many  modern  text- 
books follow  this  principle  of  selective  enhancement  and  simplification. 
(Illustrations  of  this  sort  require  a special  skill,  guided  both  by  technical 
knowledge  and  communication  insight.) 

Caricatures  should  probably  be  used  after  the  student  has  been  initially 
exposed  to  samples  of  the  "real"  object;  they  should  be  discontinued  as 
the  student  becomes  able  to  reliably  and  automatically  make  the  Identifica- 
tion (along  with  identifications  of  the  other  objects  in  the  set  to  be 
learned.)  It  is  possible  that  the  use  of  an  artificial  caricature  in  train- 
ing may  establish  a learning  and  recall  strategy  for  the  student  which  he 
will  carry  into  operational  activities.  Thus,  it  could  serve  as  a supplement 
to  learning  the  identifications  of  entities  in  the  real  work  situation. 
Learning  identifications  in  the  actual  work  environment  is  ideal  only  if 
the  right  conditions  prevail  --  individualized  tutoring  on  object-name  and 
name-object  with  immediate  knowledge  of  results  and  correction  as  necessary, 
continued  for  the  many  trials  that  lead  to  dependable  recall. 

Contextual  cues  for  identi fl cation.  The  method  used  to  learn  identifications 
can  make  their  recall  highly  dependent  on  contextual  cues.  Training  in 
context  is  efficient  and  effective  when  those  contexts  are  present  in  the 
work  situation.  Keep  in  mind,  however,  that  a high  degree  of  practice  tends 
to  lead  to  minimum  sampling  of  presented  cues  before  an  "identification"  is 
made.  If  after  such  training  and  experience,  the  operator  must  make  the 
identification  in  a strange  context,  he  may  be  unable  to  do  so  reliably. 

Thus,  if  Identifications  must  be  made  in  a large  and  somewhat  unpredictable 
range  of  contexts,  practice  on  absolute  (as  contrasted  with  relative  or  con- 
textual) identification  is  prescribed.  Identification  should  be  learned  in 
a minimum  context  (excluding  secondary  matters,  except  for  other  members  of 
the  set  to  be  discriminated  from  each  other),  and  in  a wide  range  of 
randomly  sequenced  conditions.  The  geologist,  archeologist,  and  detective 
share  this  kind  of  learning  and  performance  problem. 

Interpretation  and  identification.  The  operator  builds  an  identification 
from  a number  of  contextual  and  inferential  cues.  An  inexperienced  diagnos- 
tician makes  a number  of  tests  and,  by  a deliberate  inferential  process, 
reaches  a diagnosis.  An  experienced  practitioner  recognizes  the  pattern  of 
symptoms  all  at  once  and  identifies  the  trouble.  In  computer  jargon,  this 
is  equivalent  to  translating  a computational  operation  into  a "pattern 
matching"  operation.  The  pattern  of  elements  is  apprehended  more  or  less 
as  a whole  and  matched  with  a pattern  in  memory  to  which  a name  is  attached. 
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Training  has  the  option  of  mimicking  the  extensive  array  of  work  experiences 
so  that  a large  proportion  of  the  neophyte’s  tasks  are  performed  by  identi- 
fication rather  than  interpretation.  Costs  of  this  kind  of  training  in- 
crease as  the  number  of  work  situations  is.  increased.  Expenditures  may  be 
justified  if  the  mission  is  highly  significant,  and  if  task  performance 
time  is  operationally  critical.  The  lunar  space  missions  are  examples; 
training  was  undoubtedly  directed  to  the  recognition  and  identification  of 
troubles,  insofar  as  conditions  could  be  anticipated,  rather  than  to  time- 
consuming  deductive  and  interpretive  processes. 

There  are  operational  tradeoffs  between  the  identification  mode  and  the 
interpretive  mode.  Spontaneous  identification  may  be  based  on  inappro- 
priate cue  sampling  and  on  irrelevant  contexts.  The  interpretive  process, 
although  slower,  is  essentially  one  of  hypothesis-forming  and  hypothesis- 
testing and  tends  to  be  more  careful  and  sometimes  more  accurate.  If  time 
permits,  and  mission  importance  justifies,  the  best  arrangement  Is  redundancy 
of  both,  using  spontaneous  identification  in  creating  the  hypothesis  and 
applying  the  interpretive  process  to  test  that  hypothesis. 
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What  does  it  mean? 

Rules  for  translating  the  symbolic  context  of 
a message  into  a reference  or  meaning,  usually 
by  the  addition  of  reference  context  from  within 
the  message  itself,  but  at  times  from  outside 
the  message. 

Interpreting  requires  response  to  a pattern  of  cues  and  applies  to  events 
or  conditions  that  go  beyond  input  data  or  symbols  as  such.  Input  data 
are  only  a part  of  the  total  information  required  for  interpretation. 

This  is  the  way  we  di fferentiate  interpretation  from  decoding. 

An  interpretation  is  an  inference  about  a condition,  state  of  affairs, 
or  source  of  data. 

Examples 

» Automatic  analysis  and  "recognition"  of  an  English  word  as  a pattern 
contained  in  the  physical  wave  form  of  an  utterance. 

• Human  conclusion  that  the  unannounced  approach  of  foreign  aircraft, 
detected  and  identified  on  radar  screens,  means  invasion  and  war. 

• Human  conclusion  that  a given  pattern  of  symptoms  signifies  that  a 
system  failure  must  be  caused  by  a programming  error  rather  than  a 
machine  failure. 

• Language  translation  from  Greek  to  English  expressions. 

Process  Variables 


• Degree  of  statistical  certainty  regarding  correctness  required  of 
the  interpretation. 

• Amount  of  redundancy  in  the  form  of  context  available  in  the  message. 

• Range  of  variability  among  elements  in  the  pattern  to  be  determined. 

• Proportion  of  irrelevant  transients  in  the  message  which  act  as 

noise  to  interpretation. 

• Number  of  elements  necessary  and  sufficient  to  match  with  a reference 
set  ("dictionary")  of  meanings  or  interpretations. 

• Number  of  alternative  meanings  or  identifications  in  the  reference 
set  available  for  the  matching  effort. 

• Opportunity  for  interpreter  to  query  message  source  for  additional 
information  for  testing  hypotheses  about  an  interpretation. 
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Discussion 


Inducing  or  projecting  a "meaning"  on  a stimulus  pattern  or  on  message 
components  in  terms  of  some  reference  context,  is  one  of  a variety  of 
thumbnail  definitions  of  interpretation.  The  essential  ingredients  are 
generally  those  of  responding  to  more  than  what  is  immediately  present 
or  perceived,  and  the  capacity  of  the  presented  cues  to  signify  sub- 
jectively a cognition  of  state  or  purpose.  Some  process  of  logical  or 
material  inference  is  implied,  and  a translation  of  a set  of  signs  con- 
ceptualized. The  translation  may  be  from  a symbolic  representation,  such 
as  text  or  diagram,  to  physical  representation  or  action.  Or,  the  trans- 
lation may  be  in  the  reverse  direction;  from  a perceived  actuality  to  a 
symbolic  representation  in  thought.  A meaning  (in  the  sense  of  inter- 
pretation) may  be  expressed  in  what  the  operator  thinks,  what  he  does,  or 
by  a combination  of  both. 

Definitions  of  this  kind  are  invariably  'incomplete,  vague,  or  trivial  be- 
cause a condition  of  awareness  is  connoted  and  objective  definitions  about 
states  of  awareness  are  likely  to  be  vague  or  trivial. 

Examples  of  interpretation  abound.  The  concept  of  the  interpretive  process 
may  be  among  those  which  are  useful  mainly  when  they  are  rooted  in  some 
relatively  definable  operational  context.  Thus,  language  interpretation, 
map  interpretation,  radar  scope  interpretation,  animal  track  interpretation  -- 
each  suggests  a fairly  specific  combination  of  environmental  input,  purpose, 
and  process  which  is  pragmatically  manageable.  Note,  however,  that  inter- 
pretation does  not  include  (psychologically)  literal  code  translation  such 
as  translating  English  text  into  Morse  code  according  to  a fixed  (to  the 
operator)  set  of  rules.  When  an  operator,  through  experience,  does  find 
and  apply  a set  of  rules  for  responding  in  some  class  of  situations,  the 
activity  becomes  either  a matter  of  decoding  or  an  identification  process 
rather  than  an  interpretive  process.  The  rules  may  be  implicit  and  the 
operator  may  not  be  able  to  articulate  them.  (This  point  will  recur  later.) 

An  interpretation  is  at  least  a subjective  technique  for  the  reduction  of 
uncertainty  presented  by  a complex  of  environmental  signs  that  do  not 
(according  to  the  operator's  programs)  directly  evoke  goal-relevant 
responses,  or  that  tend  to  evoke  inconsistent  or  conflicting  responses.  If 
we  may  anthropomorphise,  it  is  as  if  the  human  abhors  ambiguity,  at  least 
in  its  subjective  state  where  ambiguity  is  translated  into  uncertainty. 
Interpretation  is  an  anxiety  reducer.  (A  second  class  of  anxiety  may 
follow  one's  knowledge  or  belief  in  knowledge  of  a set  of  conditions  -- 
that  is,  what  to  do  about  it  --  but  that  problem  takes  us  into  decision- 
making.) An  insight  is  an  interpretation,  and  the  phenomena  and  strategies 
associated  with  the  psychological  literature  on  insight  are  directly  appli- 
cable here. 

The  analyses  of  decision-making,  interpretation,  and  identification  come 
together  and  overlap  somewhat.  (See  earlier  subsection.  Identify.)  With 
repeated  experiences,  adaptive  processes  in  the  operator  tend  to  change  what 
at  one  time  is  an  interpretive  process  into  an  identification  process.  This 
change  does  not  invalidate  the  distinction  between  the  two  processes,  nor 
the  value  of  separate  but  overlapping  strategic  variables  for  each  of  them. 
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The  overlap  does  make  it  difficult  for  judges  who  must  try  to  distinguish 
one  process  implication  from  another  in  examining  a task  or  collection  of 
related  tasks.  The  semantic  distinction  in  itself  should  not  be  con- 
sidered important  except  to  suggest  operations  and  strategies  of  relevance 
to  task  performance,  task  design,  selection,  training,  and  so  on. 

Behavioral  Tendencies 


As  in  other  behavioral  processes,  repetitions  in  a task  context  that  require 
a class  of  interpretations  (such  as  language  translation  or  determining  the 
intent  of  an  opponent  in  a given  kind  of  war  game),  lead  to  various  regu- 
larities of  sequence,  standardizations  of  processing  format,  selection,  and 
prioritization  of  cues  which  may  potentially  be  attended  to  in  a given 
situation  requiring  an  interpretation.  These  may  include  cues  available 
for  "testing"  an  interpretation  subsequent  to  making  the  interpretation. 


By  definition,  all  of  the  cues  necessary  and  sufficient  for  arriving  at  a 
conception  of  meaning  in  terms  of  a task  situation,  are  not  immediately 
present  or,  if  present,  are  not  given  immediately  to  apprehension.  There 
is  opportunity  and  need  for  the  operator  to  project  information  into  what 
is  given  objectively.  An  interpretation  is , therefore,  an  hypothesis,  which 
may  be  held  with  varying  degrees  of  tentativeness  or  certainty.  The  hypoth- 
esis may  be  more  or  may  be  less  valid  according  to  some  criterion  independent 
of  the  operator's  concept. 

The  translation  of  a foreign  language  affords  a convenient  example  of  pro- 
gressive change  in  processing  with  practice.  The  novice  begins  by  trans- 
lating words,  more  or  less  literally,  one  word  at  a time.  As  he  progresses 
in  vocabulary  and  syntax,  he  translates  a phrase  as  a unit,  buffering  in 
short-term  memory  the  several  words  of  the  phrase  that  he  reads  or  hears 
until  the  phrase  unit  is  more  or  less  completed.  With  still  more  practice, 
he  tends  to  use  the  clause  or  similar  "unit  of  thought"  as  the  context  for 
interpreting  a meaning  of  the  te*t.  Thus,  he  becomes  able  to  "translate" 
words  that  are  not  entirely  in  working  vocabulary  because  of  their  context. 

If  he  is  translating  from  German  into  English,  he  must  acquire  a somewhat 
different  structural  pattern  (in  short-term  memory)  for  retaining  elements 
in  the  symbolic  pattern  which  yield  a meaning.  That  is,  he  learns  different 
message  formats  from  those  he  is  accustomed  to  in  English.  Ultimately,  he 
may  be  able  to  "read  for  sense"  without  recasting  the  syntactic  structure 
of  the  language  he  is  reading  into  that  of  his  native  language. 

Quite  literally,  the  student  translator  is  building  up  an  hypothesis  of 
the  meaning  of  a sentence  and  paragraph  in  the  course  of  reading  its  elements. 
The  test  of  the  hypothesis  is,  "Does  the  sentence  make  sense  with  respect  to 
some  kind  of  reference  to  the  context  and  the  reader's  experiences  with 
sensible  discourse?"  This  test  may,  unfortunately,  be  omitted  in  many 
interpretive  acts  of  any  kind  when  the  operator  is  preoccupied  with  literal 
transcription  of  the  component  symbols.  Perhaps  this  is  why  one  of  the 
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tests  of  sophistication  in  a foreign  language  is  the  ability  to  understand 
a joke  told  in  that  language,  which  would  elicit  amusement  if  told  in  the 
native  tongue.  Jokes  usually  require  the  perception  of  an  incongruity 
within  a pattern  of  references. 

Let  us  now  focus  on  an  outline  of  the  factors  in  the  learning  process 
associated  with  interpretation  in  a class  of  environments. 

• Cue  minimization.  With  continued  practice,  an  hypothesis  (conclu- 
sion or  interpretation)  is  developed  from  fewer  and  fewer  cues. 

• Standard  test  sequencing.  Depending  on  frequency  of  failure  data 
("hypothesis  is  incorrect"),  standard  test  sequences  in  the  examina- 
tion of  cues  are  set  up  in  order  to  test  a tentative  hypothesis,  or 
to  develop  it  in  stages.  Tests  that  fail  to  make  differentiations 
tend  to  be  eliminated. 

• Short- tenn  memory  content.  The  number  and  kind  of  cues  that  are 
clumped  in  short-term  memory  before  reaching  an  hypothesis  become 
standardized.  For  example,  a careful  translation  of  a statement 
made  in  Chinese  into  English  may  not  be  initiated  until  major 
segments  of  a sentence  have  been  completed,  or  perhaps  until  the 
entire  sentence  has  been  heard  or  read.  The  sentence  becomes  the 
unit  or  pattern  of  cues  from  which  the  "interpretation"  of  meaning 
is  based,  and  this  concept  of  the  meaning  is  what  is  coded  by  the 
translator  into  English. 

• Priori ti zi ng  of  cue  types . Some  cues  become  more  important  than 
others  in  establishing  an  interpretation  or  a concept  of  meaning. 

The  priorities  are  likely  to  be  highly  individual  because  they  are 
linked  to  code  hierarchies  in  long-term  memory  that  are  specific 

to  the  individual's  experience,  and,  possibly,  also  to  his  internal 
preference  for  one  kind  of  code  over  another  (e.g.,  auditory  pre- 
ferred over  visual,  spatial,  or  kinesthetic). 

• Influence  of  recency.  Recent  events  in  the  operator's  experience 
tend  to  dominate  or  at  least  influence  interpretations,  especially 
where  the  cues  are  ambiguous  or  incomplete.  In  other  words,  the 
interpretation  is  governed  by  the  phenomena  subsumed  under 
behavioral  "set". 


# 
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• Dropping  out  of  cognitive  mediating  operations.  With  high  degrees 
of  successful  practice  with  the  interpretive  content  in  a given 
task,  the  interpretive  mode  --  in  terms  of  the  active  processes  of 
implication  --  disappears  and  merges  into  an  "identification"  type 
of  response.  In  other  words,  implicit  verbalization  and  image-making 
become  reduced  as  the  operator's  repertoire  of  standard  response  to 
standard  situation  is  expanded  (even  though  the  situation  is  complex). 

• Stereotyped  repertoire.  Differentiations  in  response,  according  to 
subtle  differences  in  the  input  messages,  may  become  reduced  to  a 
relatively  small  stereotyped  repertoire.  This  is  most  likely  to 
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occur  if  the  operator  must  interpret  under  time  pressure  and 
according  to  a quantitative  criterion  rather  than  a qualitative 
criterion.  Stereotypes  may  crystallize  around  the  verbal  code 
repertoires  that  are  in  the  task  vernacular.  The  reduction  of 
alternatives  from  which  selections  are  made,  and  the  standardiza- 
tion of  the  cues  for  selecting  an  alternative,  clearly  are 
reductions  in  data  processing  complexity. 

Task  Strategies 

Task  strategies  aimed  at  improving  the  interpretation  process  are,  for  the 
most  part,  aimed  at  avoiding  or  minimizing  the  behavioral  liabilities  already 
described.  More  positive  and  specific  task  strategies  are  likely  to  depend 
on  the  specifics  of  the  task(s)  and  of  the  environment(s)  to  be  interpreted. 

In  general  terms,  these  will  be  the  cliches  of  rational  as  contrasted  with 
impulsive  behavior  --  form  hypotheses  tentatively;  accumulate  evidence 
through  extensive  sampling;  correct  hypotheses  accordingly;  test  the  conclu- 
sion by  reference  to  some  larger  pattern  or  context  in  which  the  samples 
of  data  appear. 

The  distinction  between  a training  technique  and  a performance  strategy  be- 
comes highly  blurred  in  these  complex  orders  of  behavior. 

Avoid  stereotypical  response.  Reducing  error  probability  in  interpretation 
follows  from  reducing  the  tendency  to  stereotypical  response.  This  avoidance 
consists  of  taking  more  perceptual  samples  of  the  situation  to  be  interpreted, 
or  larger  groupings  of  samples  to  be  inspected  as  a pattern  (thus  getting 
more  context),  and  reducing  the  tendency  to  semantic  oversimplification. 

That  is,  avoid  the  tendency  to  impose  a name  on  the  phenomenon  without  check- 
ing further  for  data  that  qualify  the  name  (as  by  subscripting  it)  or, 
qualify  the  interpretation  indicating  that  extrapolation  and  generalization 
or  conclusions  based  on  the  data  examined  would  be  risky. 

Offsetting  the  spontaneous  behavior  tendencies  requires  either  continuously 
active  motivation  by  the  operator  to  do  so,  or  the  control  of  the  program- 
ming of  work  samples  to  which  the  operator  is  exposed,  and  giving  him 
immediate  error  feedback  when  he  makes  mistakes.  The  latter  is  done  in  train- 
ing. It  may  be  difficult  in  operations  unless  training  activities  can  be 
interspersed.  Thus,  what  tends  to  happen  is  that  the  operator  may  become 
fast  and  reliable  in  interpreting  standard  situations,  but  unreliable  for 
interpreting  novel  circumstances  because  of  his  stereotypical  behavior. 

Consider  the  interpretation  to  be  tentative.  In  amplification  of  the  fore- 
going  point,  the  operator  should  be  trained  (and  motivated  to  continue  after 
training)  to  treat  any  interpretation  as  a tentative  hypothesis  that  can  be 
confirmed  only  to  a degree  by  (a)  more  data  samples,  and  (b)  testing  other 
hypotheses  against  wider  contexts  of  the  data  presented.  One  is  tempted  to 
say,  "Check  the  conclusion  against  common  sense."  Failure  of  the  conclusion 
to  fit  "common  sense"  is  not  sufficient  basis  for  denying  it,  but  should  be 
a basis  for  examining  more  data  if  they  are  available,  or  for  making  the 
interpretation  very  tentative. 
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The  capacity  for  this  kind  of  semantic  sophistication  may  involve  sub- 
stantial individual  differences  that  may  counter-balance  even  lengthy  and 
dedicated  training. 

Optimize  chunk  of  information  for  interpreting.  A significant  performance 
strategy  would  aim  at  optimizing  the  unit  of  aata  that  is  sampled  in 
accordance  with  the  unit  of  what  is  being  interpreted.  Some  optimum  or  at 
least  workable  unit  of  pattern  should  be  selected.  In  language  translation, 
this  unit  would  be  the  phrase,  possibly  the  sentence.  Interpretation  of 
the  intent  behind  a political  maneuver  may  not  have  any  clear  unit  of 
pattern  structure,  so  that  the  following  may  be  required. 

Identify  key  variables.  Identify  key  variables  and  search  for  data  samples 
of  these  key  variables  as  a foundation  for  the  interpretation  (hypothesis). 
This  strategy  assumes  that  interpretations  are  made  in  some  reasonably  well- 
defined  situational  context.  It  may  be  a rather  explicit  procedural izati on 
of  the  steps  to  be  taken  in  getting  data  samples  to  be  interpreted.  If  key 
variables  are  examined  by  the  operator  in  a consistent  order,  he  will  be 
formatting  the  activity.  To  the  extent  that  the  format  he  uses  is  complete 
(and  to  the  extent  that  the  complete  format  is  maintained  in  task  behavior), 
the  chances  for  valid  interpretation  are  increased. 

A strategic  delay  in  action.  A strategic  action,  whereby  the  operator  de- 
liberately imposes  delay  between  the  sampling  of  data  and  deriving  an 
hypothesis  or  conclusion  from  that  data,  is  used  to  offset  a behavioral 
liability  and  has  performance  validity.  "Look  again  before  you  leap." 

"Think  before  you  act."  It  is  the  pause  of  critical,  rational  reflection 
before  commitment.  The  length  of  the  delay  may  be  very  brief  (e.g.,  a few 
tenths  of  a second).  Imposing  the  delay  for  critical  examination  is 
probably  context-dependent  when  learned.  Thus,  the  college  professor  may 
pause  when  offered  a conclusion  about  an  experiment  by  a colleague,  but 
reflexively  respond  immediately  to  a student's  remarks  that  he  thinks  the 
tests  are  unfair  because  ...  How  the  delay  is  filled  is  what  is  important  - 
merely  counting  to  ten  before  responding  is  beside  the  point.  The  probable 
value  of  the  cognitive  interval  to  task  performance  is  that  it  tends  to 
avoid  spontaneous  stereotype  response  to  the  stimulus  of  the  immediate 
present,  and  to  substitute  a broader  reference  base  of  information  in 
arriving  at  and  testing  the  hypothesis  or  conclusion  which  is  the  "inter- 
pretation". This  is  a learned  strategy  in  coping  with  input  information 
(as  it  also  is  with  emotional  impulses),  and  is  readily  unlearned  in  task 
environments  in  which  unusual  patterns  requiring  interpretation  are  rare. 

Watch  for  qualifying  cues.  In  many  task  situations,  a correct  interpreta- 
tion  (as  distinguished  from  many  acts  of  "recognition"  or  identification 
which  depend  on  the  similarity  underlying  various  cue  patterns)  depends  on 
the  perception  of  differences  between  the  instance  at  hand  and  the  reference 
set  of  instances.  (Thus,  the  statement,  "Orientals  tend  to  be  more  polite 
than  Occidentals",  may  be  incorrectly  interpreted  as  "All  Orientals  are 
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always  more  polite  than  any  Occidentals  at  any  time."  This  interpretation 
is  a dangerous  guide  to  behavior  and  is  subject  to  disillusionment  and  re- 
jection of  the  correctly  stated  assertion  --  assuming  it  is  actually  true  -- 
and  a denial  of  the  credibility  of  the  source  of  the  assertion.)  The 
operator's  failure  to  detect  cues  to  critical  di fference(s)  in  the  pattern 
to  be  interpreted  makes  progressive  improvement  in  his  ability  to  interpret 
a class  of  situations  difficult  or  impossible. 

It  is  often  necessary  for  a subordinate  to  interpret  the  orders  he  receives 
from  his  superiors.  Failure  to  distinguish  between  the  following  statements, 
for  example,  is  likely  to  lead  to  trouble.  "I  want  you  to  use  your  initiative 
but  you  will  be  accountable  for  a high  level  of  judgment."  "I  want  you  to 
use  your  initiative  even  though  it  may  take  a lot  of  courage  to  carry  out 
the  action."  The  first  admonishes  daring  and  implies  conservatism  and 
caution.  The  second  reverses  these  priorities. 

Each  class  of  task  and  environment  probably  has  its  own  sets  of  key  qualifying 
cues  which  must  be  learned  and  applied  for  the  operator  to  gain  interpretive 
expertise.  These  cues  rarely  enter  the  classroom  or  instructional  syllabus. 

If  they  are  transmitted  at  all,  they  are  likely  to  be  passed  on  informally 
from  the  expert  to  the  novice  in  the  work  situation.  Such  instruction  has 
the  advantage  of  direct  relevance,  but  is  subject  to  liabilities  of  verbal 
instruction. 

Predictive  interpretation.  The  purpose  of  many  kinds  of  interpretation  is 
to  make  a prediction.  Meteorology  is  a technique  for  interpreting  data 
about  past  and  present  weather  conditions  in  order  to  predict  weather  in  a 
given  locale.  Interpreting  the  behavior  of  a motorist  at  an  intersection 
is  for  the  purpose  of  predicting  what  he  is  going  to  do  as  it  may  affect 
your  own  plans  and  maneuvers.  Observation  of  samples  of  the  behavior  of  a 
maintenance  technician  who  is  checking  out  your  aircraft  has,  as  its 
practical  purpose,  your  prediction  as  to  whether  you  will  have  trouble  with 
your  aircraft  in  flight. 

Predictions  are  inherently  more  unreliable  than  the  interpretation  of  a 
present  state  of  affairs.  It  should  follow  that  a broader  range  of  samples 
should  be  taken  and  assessed  in  interpreting  the  future  than  interpreting 
the  present.  Prediction  of  the  future  implies  a date  at  which  something 
will  happen;  trend  data  must  be  sampled  in  order  to  make  the  temporal  pro- 
jection. 

Task  strategies  need  to  be  developed  for  estimating  rates  of  change  and 
projecting  points  in  time  when  various  events  will  intersect.  When  will 
the  baseball  thrown  by  pitcher  X be  over  home  plate  in  batting  range?  If 
the  man  up  is  weli  instructed,  he  will  have  learned  to  take  a systematic 
series  of  samples  of  the  ball  en  route  to  the  plate.  If  the  sampling 
series  is  systematic  with  respect  to  standard  time  intervals,  he  has  an 
experience  reference  against  which  to  evaluate  present  data  on  the  ball's 
motion.  (For  the  moment  we  are  disregarding  the  problem  of  the  direction 
of  the  ball's  trajectory  as  if  this  were  a separate  problem,  which  psycho- 
logically it  probably  is  not.)  If  the  rate  is  constant,  two  observations 
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of  relative  position  are  enough  theoretically  to  predict  its  moment  of 
arrival,  assuming  a constant  distance  between  the  pitcher  and  batter  and 
fixed  distances  at  which  the  position  of  the  ball  is  observed. 

The  same  principle  of  time  sampling  is  used  by  rational  managers  in  assess- 
ing whether  a project  will  be  completed  by  a given  date.  Of  course,  manage- 
ment also  is  aware  of  the  dynamic  properties  of  work  and  workers  on  a given 
kind  of  project,  and  thus  can  project  where  changes  in  relative  rate  are 
likely  to  occur.  Samples  are  taken,  not  randomly,  but  at  certain  key  points 
in  progress.  These  key  points  are  learned  through  a background  of  experience. 
The  professor  in  a graduate  school  can  predict  whether  the  student's  disserta- 
tion will  be  completed  within  a year  by  considering  the  data  on  which  the 
student  defines  the  problem  and  the  date  on  which  the  first  datum  is  collected. 

Closely  Related  Topics 

"Interpret"  is  closely  related  to  Identify  and  Decision  Making. 


# 
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Choosing  a response  to  fit  the  situation 

In  a broad,  inclusive  sense,  decision  making  consists 
of  organization  and  processing  of  the  following  classes 
of  information:  need  or  want  variables  and  situation 

variables  (two  classes  of  variables  that  characterize 
the  "problem  situation");  response  alternatives;  and 
(possibly)  some  strategy,  policy,  or  rule  for  matching 
the  implications  of  a response  alternative  or  resource 
available  to  the  profile  of  needs  and  situation. 

Choices  are  usually  made  in  a condition  of  some  uncertainty  about  actual  situa- 
tions, criterion  or  need,  and  uncertainty  as  to  how  a given  choice  will  work 
out. 

In  mechanized  decision  making,  the  response  alternatives  are  exclusive  of  each 
other,  and  the  classes  of  data  and  data  content  are  sufficient  for  operating 
the  rule  which  selects  the  response  from  the  response  repertoire.  The  rule  may 
extend  to  handling  probabilistic  data,  or  even  include  random  selection  devices 
(in  order  to  break  action  deadlocks).  The  designer  of  the  rule  includes  an  im- 
plicit definition  of  "purpose"  in  the  formulation  of  the  rule  and  principles 
for  selecting  a response  to  a situation. 

Purpose  is  rarely  simple  in  real-life  human  decision  making,  where  substantial 
commitments  are  made  by  putting  a choice  into  action.  Intentions  and  wants  are 
numerous,  interactive,  and  tend  to  remain  in  flux.  Ordinarily,  one  must  trade 
off  one  kind  of  objective  with  another  objective,  for  example,  low  cost  versus 
high  quality  or  dependability.  This  is  a flux  situation  (like  a stewpot)  rather 
than  a matrix  formulation  of  values  arranged  systematically  according  to  magni- 
tudes of  x and  y arguments. 

Human  decisions  in  real  life  tend  to  be  characterized  by  uncertainty  about  many 
things:  what  one  wants;  how  much  one  will  pay  for  what  he  wants  in  terms  of 

penalties  or  loss  of  resource;  what  the  situation  really  consists  of,  or  even 
what  one  should  know  about  the  situation  in  order  to  define  "the  problem;"  the 
range  of  responses  from  which  choice  may  be  relevant;  implications  of  the  re- 
sponse in  terms  of  commitment;  or  probability  of  success  or  ensuing  entangle- 
ments. The  imposition  of  probability  estimating  on  correctness  of  interpreta- 
tion — what  the  facts  actually  are  below  the  presented  facade  — or  on  response 
capabilities  (for  success  or  failure)  may  or  may  not  materially  contribute  to 
the  rationality  of  the  choice-making  operation. 

As  an  example  of  the  interplay  of  these  factors,  recall  some  major,  non-impulsi ve 
purchase  such  as  that  of  an  automobile  or  a dwelling.  Or,  consider  a decision  to 
overtake  and  pass  an  automobile  on  a wet,  unfamiliar  two- lane  highway  with  a 
curve  looming  ahead. 

The  actual  operation  of  making  a rational  choice  implies  that  all  of  the  avail- 
able (and  hypothesized)  information  is  taken  into  account.  In  come  cases,  deci- 
sion situations  are  formally  structured  so  that  the  process  can  be  segmented 
and  each  segment  treated  serially.  The  so-called  "decision  tree"  is  an  example 
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of  segmenting  a complex  decision  into  units  that  can  be  treated  one  at  a time. 

In  operational  situations,  however,  the  operator's  rational  choice  tends  to  be 
limited  by  what  he  can  hold  in  mind  at  one  time  and,  thus,  is  subject  to  the 
limits  of  his  working  or  short-term  memory.  This  limitation  implies  a psycho- 
logical screening  process.  The  filter  in  this  screen  may  consist  of  preferences, 
stereotypes,  projections,  dramatized  cues  and  wishes  --  the  variety  of  factors 
which  we  have  examined  under  the  rubric  of  Interpret.  In  fact,  the  definition 
of  the  process  of  interpreting  precisely  fits  the  "situation  analysis"  aspects 
of  decision  making.  Interpreting  is  normally  a part  of  decision  making. 

The  limitations  of  working  memory  for  holding  information  makes  it  necessary 
to  impose  simplifications  in  what  is  thought  about  and  short  cuts  in  "logical" 
processing.  Constructive  thinking  may  consist  in  large  part  of  the  examination 
of  a series  of  facets  of  the  situation  and  sifting  out  those  factors  which  are 
marginal  context  from  those  which  are  central  to  the  choice.  One  major  factor 
in  effective  skill  in  complex  choice-making  may  consist  of  a pattern  of  learn- 
ing experiences  which  enable  quick  separation  of  marginal  from  central  factors. 

A second  factor  in  decision  making  skill  would  consist  of  the  functional  in- 
crease of  the  information  which  working  memory  can  hold  in  the  process  of  reach- 
ing a choice.  The  subsection  on  decision  making  task  strategies  discusses 
hypotheses  for  ordering  the  assimilation  of  information  content  about  a given 
situation  so  as  to  facilitate  its  retention  in  working  memory. 

As  a person  works  on  a job,  he  tends  to  separate  decisions  which,  from  his  point 
of  view,  have  marginal  consequences,  from  those  with  dramatic  consequences.  He 
tends  to  make  marginal  decisions  by  impulse,  with  little  or  no  reflection.  The 
Impulse  may  be  founded  on  an  extended  series  of  decisions  and  consequences  in 
his  experience,  so  that,  in  effect,  he  "recognizes"  the  proper  choice  from  the 
data  presented  or  inferred.  Or  he  may  become  adept  at  mental  coin-flipping  be- 
cause he  does  not  expect  that  further  thinking  or  further  information  (if  avail- 
able) is  likely  to  improve  the  quality  of  the  decision  he  can  make  in  that  situ- 
ation. (This  assertion  assumes  the  operator  is  not  clinically  disturbed.) 

With  practice  and  a subjectively  reasonable  batting  average,  the  operator  tends 
to  develop  format  characteristics  for  classes  of  decision  that  continue  to  re- 
quire reflection.  The  format  is  much  like  an  implicit,  standardized  question- 
naire of  conceptualized  variables.  If  possible,  the  decision  maker  structures 
his  inquiry  so  that  the  questions  are  answered  in  a standard,  habitual  sequence. 
The  ordering  of  the  content  may  be  based  on  the  sequential  ordering  of  data 
variables  from  the  information  source,  or  the  ordering  may  be  based  on  the 
ability  of  the  decision  maker  to  control  the  sequence  of  inquiry.  The  exper- 
ienced decision  maker  also  tends  to  structure  his  inquiry  by  acquired  criteria 
of  relevance.  (Relevance  is  determined  by  the  answer  to  a question:  Does 

this  kind  of  information  contribute  to  the  selection  of  a response  I am  able  to 
make?)  This  filtering  process  obviously  simplifies  the  amount  of  information 
which  the  decision  maker  must  process.  Although  psychologically  desirable,  this 
tendency  can,  on  occasion,  lead  to  errors  in  decision  making  that  are  serious 
from  an  operational  standpoint. 

Just  as  many  repetitions  of  the  interpretive  process  in  a class  of  situations 
can  result  in  reducing  the  processing  activity  to  an  "identification"  of  an 
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individual  "member"  in  that  class,  so  too  can  the  repetition  of  deci sion  making 
activities  (originally  requiring  both  interpretation  and  response  selection 
through  vicarious  testing  of  response  implications)  lead  to  virtually  procedural 
response  at  the  near  automatic  level  of  behavior.  This  process  is  behavioral ly 
adaptive  according  to  the  processing  econonjy  of  the  human  as  an  adaptive  system. 

On  occasion,  internal  adaptation  of  this  sort  has  operational  liabilities. 

Where  such  liabilities  can  be  anticipated,  the  training  process  should  attempt 
to  inhibit  the  progress  of  the  reflective  process  towards  mere  automatic  re- 
sponse. But  arresting  this  development  has  several  penalties:  longer  delays 

between  the  psychological  onset  of  the  problem  situation  and  the  action  response 
(thinking  takes  time);  diminished  ability  to  attend  to  other  things  going  on  at 
the  same  time  (thinking  uses  up  channel  capacity). 

Most  of  these  factors  are  itemized  in  the  following  outline  of  behavior  ten- 
dencies . 

Behavior  Tendencies 

Short  cuts  can  be  expected  in  sampling  and  processing  all  classes  of  information 
cited  in  the  definition  with  which  we  began  this  section  on  decision  making. 

Standardization  of  goal  variables.  Need  variables  or  goal  variables  become 
standardized  and  formalized  and  may  eventually  be  dropped  from  consideration 
when  the  decision  operation  is  reduced  to  a procedure  or  habitual  formula. 
Short-term  goals  tend  to  predominate  over  longer  term  goals.  Some  goal  vari- 
ables may  be  given  inordinate  priority  over  others,  despite  unhappy  experiences 
arising  from  this  neglect  (e.g.,  deciding  to  have  those  additional  drinks  despite 
the  inevitability  of  tomorrow's  hangover  or  tonight's  traffic  hazards). 

Situation  variables.  Restricted  sampling  of  the  situation  variables  tends  to 
occur  as  decisions  in  the  subject  domain  become  habitual  and  the  internal  decision 
format  is  stabilized. 

Response  alternatives.  There  tends  to  be  restricted  testing  of  the  implications 
of  response  alternatives,  and  limited  searching  for  new  ones.  There  is  a ten- 
dency to  over-general i ze  successful  and  unsuccessful  responses,  depending  on 
prevailing  mood. 

Rationalizing  choice  policy.  The  operator  tends  to  rationalize  that  a policy 
or  principle  has  been  followed  after  a choice  has  been  made,  rather  than  as  a 
rationale  and  guide  to  choice  selection. 

Formatted  structure.  Repetitive  decision  making  as  part  of  a job  dealing  with 
more  or  similar  classes  of  situations  tends  to  become  formatted.  The  same  set 
of  variables  is  examined  in  the  same  sequence.  Fewer  variables  in  the  problem 
are  considered,  unless  the  decision  maker  works  from  an  objective  checklist 
that  requires  him  to  fill  in  data  in  each  category.  Even  the  use  of  a checklist 
can  become  a ritual  in  which  only  a restricted  set  of  data  is  treated  as  rele- 
vant. The  range  and  variety  of  response  choices  tends  to  shrink  radically  to 
a few  prepotent  favorites.  (Observers  may  call  such  a decision  maker  "preju- 
diced.") The  cognitive  operations  of  weighing  and  thinking  through  the  impli- 
cations of  situation,  needs,  and  alternatives,  tend  to  be  reduced  and  the  choice 
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tends  to  be  elicited  automatically.  These  outcomes  may  be  positively  adaptive 
to  performance  demands  if  a limited  set  of  patterns  is  frequently  repeated  and 
the  response  repertoire  and  effectiveness  of  respective  responses  does  not 
change. 

In  many  cases,  decisions  must  be  made  about  events  taking  place  in  the  future, 
which  means  there  is  some  degree  of  uncertainty  regarding  conditions,  priority 
of  needs,  availability  of  resources  to  implement  a choice.  A conservative  stra- 
tegy or  policy  will'  hedge  on  commitment  or  lean  toward  choices  that  have  tradi- 
tionally been  successful  and  never  disastrous.  A more  daring  policy  may  pro- 
ject optimism  into  the  uncertainty  and  commit  more  dangerously. 

Behavioral  strategy  in  dealing  with  uncertainty  is  a function  of  individual 
personality  variables,  mood,  and,  of  course,  the  interpretation  of  a pattern 
of  experiences. 

Task  Strategies 

Task  strategies  proposed  in  this  section  are  restricted  to  those  intended  to 
reduce  the  liabilities  and  limitations  of  general  behavioral  tendencies  in  de- 
cision making  — where  these  tendencies  do  constitute  liabilities.  The  propo- 
sals will,  therefore,  seem  general  rather  than  specific.  The  application  of 
some  may  seem  inconsistent  with  others,  but  this  is  the  nature  of  the  tradeoffs, 
compromises,  and  qualifications  which  make  up  a complex  human  skill. 

The  central  idea  of  the  following  task  strategies  is  the  structuring  (not  re- 
duction) of  a class  of  decisions  in  a job  to  a procedure.  This  is  not  neces- 
sarily equivalent  to  making  the  decision  process  altogether  mechanical,  although 
where  practicable,  it  is  likely  to  be  desirable.  The  aspects  of  the  procedure 
are  intended  to  control,  by  a combination  of  task  design  and  training  regimen, 
the  difficulties  which  the  novice  has  in  a class  of  complex  decision  situations. 

In  organizational  practice,  this  is  commonplace.  A format  is  eventually  pre- 
pared for  decision-making  situations  encountered  many  times,  perhaps  as  routine 
operations  in  an  office.  Consider  an  admissions  office  in  a college  where  de- 
cisions are  made  to  accept  or  not  accept  applicants.  A standard  questionnaire 
is  designed  to  cull  information  about  the  applicant  on  a set  of  relevant  vari- 
ables. Questionnaire  data  may  be  supplemented  by  interview  data;  the  trained 
interviewer  tends  to  work  from  an  implicit  or  explicit  check  list  of  factors 
which  are  an  extension  of  the  questionnaire.  The  staff  also  develops  a more 
or  less  well-defined  set  of  requirements  for  admission.  These  requirements,  of 
course,  may  be  modified  by  circumstances.  The  criterion  variables  for  accept- 
ance should  approximate  the  variables  in  the  questionnaire  which  describe  the 
applicant. 

Thus  far,  the  collection  of  data  relevant  to  the  decision  has  become  "well- 
structured."  As  a result  of  historical  data  and  experience,  "standard  opera- 
ting procedures"  can  be  propounded.  The  data  describing  the  case  can  be 
ordered  in  a sequence  to  match  the  "model"  of  acceptance  or  of  rejection,  there- 
by reducing  what  the  human  decision  maker  has  to  bear  in  mind  at  one  time.  He 
stores  it  in  a "message  format". 
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The  process  for  reaching  a choice  may  now  begin.  Factors  in  the  situation  that 
match  equivalent  factors  in  the  acceptance  model  can,  henceforth,  be  generally 
ignored.  The  decision  policy  may  make  rejection  (an  answer  of  "no")  automatic 
if  some  subset  of  negative  factors  is  not  matched.  (Females,  for  example,  used 
to  be  automatically  rejected  if  they  applied  to  a boys  school.)  Some  factor(s) 
may  be  given  priority  and  outweigh  some  deficiency,  e.g.,  athletic  ability  may 
be  given  priority  over  the  applicant's  overall  academic  standing.  A high  value 
in  one  desirable  variable  is  traded  against  a low  value  in  another  desirable 
variable. 

Clearly,  although  this  process  is  not  mechanized,  it  can  become  well -organized 
through  task  design.  When  a mass  of  information  becomes  organized,  the  deci- 
sion maker  can  better  cope  with  a mul ti -variate  problem,  and  better  approxi- 
mate a rational  model  even  with  semi-rational  or  non-rational  information  such 
as  beliefs,  values,  and  preferences,  as  well  as  subjective  estimates  of  proba- 
bility. 

Operations  research  is  a technical  example  of  the  attempt  to  formalize  decision 
structure,  but  with  the  addition  of  metrics,  quantitative  assumptions,  and  ex- 
plicit policy  rules  for  optimizing  criterion  variables.  According  to  critics, 
practical  difficulties  with  it  in  some  complex  situations  have  arisen  from  in- 
appropriate assumptions  and  a reluctance  to  allow  the  human  to  interact  intelli- 
gently with  the  problem.  Other  severe  problems  arise  from  having  to  treat  a 
single  variable  of  purpose  or  utility  at  a time,  and/or  from  questionable  assump- 
tions in  converting  multiple  purposes  into  a single  "utility"  scale. 

The  following  kinds  of  format  operations  are  intended  to  facilitate  the  learning 
of  decision  making  in  a given  task  environment.  All  issues  are  simplified,  in 
many  cases  oversimplified. 

Establish  the  range  of  response  options.  Determination  of  the  range  of  responses 
from  which  a particular  decision  maker  may  properly  select  is  one  facet  of  de- 
fining the  man's  job.  It  is  the  bounding  of  lines  of  authority.  In  some  cases, 
as  In  the  case  of  the  college  admissions  officer,  this  may  be  relatively  easy  to 
do;  in  other  cases,  as  in  the  case  of  the  leader  of  an  exploratory  platoon,  such 
boundaries  may  be  practically  impossible  to  predict.  Where  a response  class  can 
be  meaningfully  specified,  criteria  of  what  information  is  relevant  and  what  is 
not,  become  established.  Information  in  a given  decision  situation  is  relevant 
If  it  contributes  to  choosing  a correct  or  adequate  response  from  the  permissible 
response  set. 

The  liability  in  limiting  the  permissible  response  set  is  that  the  decision  maker 
is  deterred  from  innovative  behavior  (e.g.,  inventing  unanticipated  new  alterna- 
tives) which  may  be  more  operationally  adaptive  and  useful  than  any  already  con- 
sidered. This  is  an  example  of  the  usual  tradeoff  between  reliability  or  effi- 
ciency versus  flexibility. 

Structure  of  hierarchy  of  decisions.  Some  complex  decisions  may  be  formatted  in- 
to  a sequential  set  of  subdecisions.  "If  decision  A is  yes,  go  on  to  B;  if  A is 
no,  consider  X.  If  B is  yes,  go  on  to  consider  C."  The  decision  to  take  a job 
with  a new  employer  might  be  sequenced  as  follows:  Is  the  pay  likely  to  be 
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acceptable,  at  least  as  a minimum?  If  the  answer  is  "no,"  there  is  little  point 
to  proceed  with  inquiry  because  the  final  decision  will  be  to  reject  the  job 
offer.  Assuming  a "yes"  answer,  the  next  question  and  decision  may  be:  Can  I 

do  the  work?  Rarely  can  an  unqualified  "Yes"  or  "No"  decision  be  made,  but  this 
is  not  always  necessary.  Rather,  if  the  decision  maker  must  make  many  decisions 
of  this  kind,  he  will  be  better  off  if  he  attempts  to  sequence  the  issues  in  a 
standardized  order.  One  outcome  of  his  experiences  will  be  a consistent  refer- 
ence pattern. 

The  decision  tree  paradigm  is  a formal  example  of  constructing  a hierarchy  of 
decisions.  It  includes  the  additional  variables  of  payoff  and  probability  of 
given  "states  of  nature." 

Develop  formats  for  acquiring  and  remembering  situation  data.  A class  of 
choice-making  situation  may  be  described  in  as  few  as  two  variables  or  as  many 
as  a score  or  more.  The  same  variables  appear  in  each  occasion  of  this  kind  of 
decision,  although  a different  value,  magnitude,  or  content  is  likely.  Left  to 
his  own  experiences  in  making  decisions,  the  decision  maker  spontaneously  devel- 
ops an  ordered  list  of  inquiry  variables  which  he  addresses  to  a given  situation 
Training  that  is  directed  to  the  formatting  and  sequencing  of  these  situational 
variables  can  accelerate  the  learning  process.  The  student  learns  more  quickly 
how  to  acquire  and  hold  in  mind  the  data  describing  the  specific  situation  con- 
fronting him.  As  a desirable  by-product,  the  ability  to  hold  this  information 
more  readily  in  mind  gives  more  mental  "work-space"  for  thinking  towards  a de- 
cision. It  also  provides  work-space  to  inquire  about  and  hold  in  mind  contin- 
gencies that  do  not  fit  the  standard  inquiry  format.  In  layman's  language,  this 
is  called  "sizing  up  the  situation." 

An  example  of  this  development  is  in  order.  A military  policeman  patrolling  a 
city  makes  decisions  as  to  whether  to  ignore  a "trouble"  situation  involving 
military  personnel,  intercede  with  verbal  advice  or  orders,  have  the  offender 
picked  up  and  returned  to  camp  without  arrest,  or  formally  arrest  and  charge 
the  offender.  He  may  examine  the  situation  for  data  responsive  to  the  following 
criteria:  Is  he  damaging  prestige  to  the  uniform  and  service?  Is  the  person 

creating  or  likely  to  create  a progressively  worse  disturbance?  Is  he  damaging 
persons  or  property  or  likely  to  do  so?  Is  he  breaking  a civil  or  military 
law?  Should  he  be  apprehended  for  his  own  good  or  protection?  Is  the  attention 
and  resource  necessary  to  intervene  in  this  case  justified  compared  to  potential 
need  to  address  more  serious  problems  that  may  arise  elsewhere?  The  appearance 
of  indecision  in  the  military  policeman  is  a liability  to  his  exercise  of  auth- 
ority. He  should  make  up  his  mind  quickly.  The  more  quickly  a complete  format 
of  inquiry  can  be  applied,  the  more  quickly  he  can  decide.  If  he  has  learned 
to  apply  criteria  automatically  in  a given  sequence,  he  can  apply  them  more 
quickly  and  more  reliably.  This  is  the  rationale  for  serializing  the  format 
for  Inquiring  about  the  status  of  a situation. 

Getting  situational  data.  The  business  executive  trying  to  make  up  his  mind 
about  a proposed  merger  and  the  would-be  vacationer  trying  to  make  up  his  mind 
whether  to  commit  a down  payment  for  Resort  A or  Resort  B,  complain  about  not 
having  "enough  data"  to  make  a reasonable  decision.  Where  choices  have  only 
probabilistic  outcomes,  there  never  is  "enough"  data  for  certainty. 
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Enough  data  for  making  a decision  has  at  least  three  kinds  of  meaning.  One  has 
to  do  with  the  subjective  sense  of  assurance  that  a given  choice  of  action  is 
better  than  known  alternatives.  On  this,  there  is  some  evidence  that  there  are 
large  individual  differences.  The  so-called  intuitives  require  relatively 
little  data  for  making  an  intuitive  leap  to  a choice;  others  continue  to  be  un- 
certain even  with  logically  complete  information  for  making  a choice.  If  this 
is,  indeed,  a fairly  cross-sectional  characteristic  of  personality,  selection 
procedures  of  one  kind  or  another  seem  desirable  for  the  decision  maker  in  a 
given  kind  of  environment. 

Enough  data  also  has  meaning  in  a rational  context.  If  a conceptual  model  of 
situational  variables  in  a class  of  decision,  such  as  initiating  a battle  cam- 
paign or  making  a merger  of  businesses,  has  been  developed,  then  absence  of 
data  about  one  or  more  of  these  variables  is,  indeed,  inadequate  data  for  making 
a choice.  In  such  a case,  two  principles  apply.  One  principle  is  that  there 
usually  are  a few  central  variables  which,  having  given  values,  should  lead  to 
peremptory  rejection  or  acceptance  of  an  hypothesis  for  the  outcome  of  a given 
action.  Data  on  these  variables  are  likely  to  be  worth  a high  price  compared 
to  data  about  other  variables.  The  principle  of  selectivity  applies.  Put  more 
effort  into  getting  key  data  for  accepting  or  rejecting  a tentative  choice. 

The  popular  literature  about  leadership  emphasizes  the  ability  to  seize  on  these 
key  variables  and  to  deal  exclusively  with  them.  It  is  not  certain  that  diff- 
erent individuals  operate  well  with  different  key  variables  in  a class  of  deci- 
sions, or  that  key  variables  can  be  abstracted  for  generalized  training.  The 
latter  hypothesis  is  worth  inquiry  where  instruction  about  important  decisions 
Is  prepared. 

A second  principle  in  gathering  Information  for  decisions  is,  in  effect,  the 
law  of  diminishing  returns.  After  some  minimum  level  of  information,  the  amount 
of  additional  useful  information  decreases  for  each  unit  of  additional  cost. 

For  his  own  choice-making  purposes,  the  seasoned  decision  maker  probably  learns 
subjectively  when  the  inflection  point  has  been  reached. 

A third  principle  for  information  gathering  also  applies.  Much  less  information 
Is  generally  required  to  reject  an  alternative  that  to  accept  it.  An  alterna- 
tive may  be  rejected  because  it  fails  to  meet  some  one  or  more  necessary  condi- 
tions. Ordinarily,  an  alternative  is  reasonably  accepted  only  when  it  fulfills 
sufficient  conditions  for  acceptance.  Generally,  sufficient  conditions  consist 
of  more  than  one  necessary  condition.  If  there  are  few  alternatives,  the  most 
attractive  Is  that  which  remains  after  the  others  have  been  rejected  on  the 
basis  of  falling  to  meet  a necessary  condition.  For  example,  a secretary  must 
be  assigned  to  a job  which  requires  nominal  typing  and  dictation  speeds,  but 
requires  fluency  in  the  Greek  language.  In  getting  data  about  American  candi- 
dates for  the  job,  the  most  efficient  first  question  to  ask  would  be:  Do  you 

speak  Greek? 

Where  a type  of  decision  Is  recurrent,  either  for  an  individual  or  an  organiza- 
tion, formats  for  the  collection  and  organization  of  situational  data  become 
standardized.  Standardized  formats  have  advantages  for  efficiency,  and  occa- 
sional penalties  of  inflexibility.  An  instructional  milieu  may  assist  the 
decision  maker  in  helping  him  to  test  the  effectiveness  and  efficiency  of  the 
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existing  data  formats  he  uses,  and  in  proposing  others  that  may  be  more  adequate 
because  (a)  they  are  more  complete,  (b)  they  are  shorter  and  more  efficient,  or 
(c)  they  are  up-to-date. 

Astute  bargainers  know  that  timing  is  important  --  the  time  at  which  new  infor- 
mation is  introduced  into  a decision  may  have  influence  disproportionate  to  its 
value  if  introduced  earlier.  Thus,  a cancer  patient  may  be  told  that  he  can 
live  for  perhaps  six  months  without  surgery,  but  with  surgery  he  has  a fifty- 
fifty  chance  of  dying  in  the  operating  room.  The  patient  vacillates.  After  an 
extended  period  of  indecision,  the  physician  may  tell  him  that  his  chances  are 
closer  to  60-40  in  favor  of  survival  after  surgery.  He  may  then  be  more  likely 
to  choose  surgery.  If,  initially,  he  had  been  told  that  his  chances  were  60-40 
in  favor  of  survival,  the  effect  might  have  been  about  the  same  as  being  told 
his  chances  were  50-50  --  indecision.  If  a genuine  deadlock  among  alternatives 
has  been  reached,  anything,  no  matter  how  minor,  which  tips  the  scale  one  way 
or  the  other,  is  logically  justified  in  swaying  the  decision.  Decisions  are 
rarely  reached  by  such  precise  weighing  of  factors. 

Processing  the  information  in  making  a choice.  The  thinking  process  for  reach- 
ing  a choice  after  reflection  is  likely  to  be  recursive.  One  set  of  factors 
after  another  is  weighed.  One  liability  of  recursive  action  is  the  likelihood 
of  circular  activity  which  leads  to  no  conclusion  at  all,  and  where,  in  final 
desperation,  the  choice  is  based  on  a kind  of  coin-flipping. 

Another  liability  is  that  because  of  information  overload,  a solution  is  sought 
by  abandoning  the  original  criteria  for  choice,  and  selecting  some  criterion 
that  has  a particularly  attractive  value  which  is  highly  visible  in  one  of  the 
alternatives.  Thus,  one  approaches  the  purchase  of  an  automobile  with  multiple 
criteria  — estimates  of  durability,  economy,  serviceability,  versatility  in 
function,  attractiveness  --  and  the  choice  is  finally  based  solely  on  which  car 
looks  most  attractive. 

A strategy  for  coping  with  either  or  both  of  these  liabilities  is  to  firmly  re- 
ject a candidate  alternative  when  the  application  of  rational  criteria  has 
shown  that  it  should  be  rejected.  This  means,  in  effect,  put  further  thought 
about  this  alternative  out  of  mind.  By  definition,  the  sophisticated  decision 
maker  is  able  to  do  this. 

In  human  affairs,  the  final  act  of  decision  is  often  an  intuitive  action.  (Call- 
ing it  intuitive  is  a confession  of  ignorance  about  the  process.)  After  a deci- 
sion has  been  announced,  many  persons  feel  obliged  to  make  their  decision  appear 
rational,  to  rationalize.  The  credibility  of  explanations  of  the  process  for 
reaching  a conclusion  may  be  dubious,  although  statements  about  factors  that  in- 
fluenced or  were  decisive  in  shaping  a choice  may  be  valid. 

The  reader  is  reminded  that  the  kind  of  decisions  we  have  been  discussing  do  not 
have  strictly  deductive  or  calculational  procedures  for  producing  answers  which 
then  are  accepted  as  given. 

If  reaching  a positive  decision  to  act  is  preceded  by  treating  the  impending 
choice  as  an  hypothesis,  remote  as  well  as  immediate  consequences  of  the  choice 
should  be  conjectured.  This  is  best  done  by  imagining  the  train  of  consequences 
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in  as  concrete  terms  as  possible.  If  the  decision  will  affect  the  interests 
and  actions  of  a group  of  subordinates,  consider  the  effects  on  various  indi- 
vidual members  of  the  group.  It  is  easier  for  a preferred  action  to  be  ration- 
alized by  thinking  of  abstractions  (which  invite  stereotypes)  than  by  samples 
of  reality.  This  is  likely  to  hold  to  some  degree  even  when  the  "samples  of 
reality"  occur  in  the  imagination. 

Exploratory  testing  of  partial  decisions.  A strategy  used  by  cautious  decision 
makers  is  to  commit  only  tentatively  to  a course  of  action,  gaining  information 
about  the  situation  relevant  to  further  coinin' tment  or  withdrawal  from  the  con- 
sequences of  tentative  action.  This  is  probably  a widespread  practice  in  our 
daily  life  as  well  as  in  institutional  life.  Its  liabilities  are  obvious.  The 
sample  data  obtained  from  a temporary  and  partial  cornin' tment  may  not  be  repre- 
sentative of  response  to  a full  commitment  extended  over  time.  If  the  explora- 
tory outcome  is  negative,  a resource  may  have  been  needlessly  lost.  If  the  out- 
come is  positive,  the  crucial  test  may  still  remain. 

These  assertions  must  be  qualified.  Where  an  entire  plan  can  be  put  into  effect, 
but  restricted  to  representative  samples  of  the  population  to  be  affected,  the 
advantages  of  exploratory  or  pilot  testing  are  clear.  It  differs  from  the  pro- 
cedure of  taking  only  one  step  at  a time  in  the  equivalent  of  a campaign,  in 
order  to  determine  the  probable  effectiveness  of  the  campaign. 

Modeling,  simulation,  and  gaming  exercises  fill  the  role  of  exploratory  testing 
of  an  hypothesis,  where  time  and  resource  permit.  These  procedures  are  exten- 
sively described  in  their  respective  literatures,  and  are  not  considered  appli- 
cable to  our  present  purposes.  What  is  germane  is  the  decision  by  one  or  more 
people  whether  or  not  to  put  into  practice  an  hypothesis  after  these  data  have 
been  collected.  A judgment  has  to  be  made  on  the  extent  to  which  these  models 
and  the  data  derived  from  them  are  sufficiently  valid  for  committing  to  a choice 
in  the  real  world. 

Comment 


In  this  section,  we  have  not  dealt  with  formal  decision-making  strategy  represented 
by  such  disciplines  as  game  theory,  operations  research,  and  simulation.  If  these 
disciplines  are  regarded  as  data-generating  tools  (rather  than  as  automated  deci- 
sion makers),  they  provide  input  to  an  individual  who  chooses  to  be  guided  or  not 
guided  by  their  output.  There  is  an  extensive  literature  on  human  behavior  in 
choice-making  situations.  One  excellent  analytic  summary  of  this  literature  was 
prepared  by  Schum  (1970)*.  The  extent  of  the  relevance  of  much  of  this  litera- 
ture, based  on  experimental  conditions  having  well -structured  problems  (e.g., 
gambling-type  games),  is  open  to  discussion,  and  perhaps  subject  to  the  inter- 
pretive skill  of  the  reader  who  is  seeking  behavioral  generalizations. 


In  DeGreene,  K.B.,  Editor,  1970.  Systems  Psychology.  N.Y.  McGraw-Hill. 


A-53 


TAEG  REPORT  NO.  12-3 


CONSTRUCT  - PLAN 

Matching  resources  over  time  to  expectations  of  needs 

Rules  and  facilities  for  predicting  what  future 
sets  of  conditions  will  occur,  what  responses  to 
make  to  them,  and  in  what  order,  so  that  given 
goals  are  likely  to  be  achieved. 

Planning  Operations 


• Predicting  the  future  — using  historical  and  present  information  to 
anticipate  which  of  a set  of  alternative  states  will  occur  at  some 
future  date  or  time  interval. 

• Exercising  priority  rules  for  determining  which  of  several  anticipated 
states  to  give  priority  of  attention. 

• Determining  the  set  of  response  capabilities  required  for  effective 
response  to  the  expected  condition  or  state. 

• Scheduling  the  resource  in  making  the  response  so  that  the  resource 
is  available  when  the  expected  condition  occurs. 

The  planner  combines  the  functions  of  predictor,  resource  selector,  and  resource 
scheduler. 

Discussion 


Planning  is  a form  of  decision  making.  But  planning  consists  of  constructing  an 
action  pattern  based  on  the  outcome  of  a number  of  choices,  rather  than  merely 
selecting  from  among  choices.  Since  a plan  deals  with  the  future  and  the  future 
Is  never  absolutely  predictable,  planning  involves  uncertainty,  hence,  choosing 
from  among  various  probabilities.  Inevitably,  there  are  planning  alternatives 
with  respect  to  goal  commitment,  goal  routes,  and  resource  selection  and  schedul 
* ing.  Instruction  in  planning  must  provide  explicit  awareness  of  these  fundamen- 
tal variables  in  the  planning  operation,  or  it  will  be  inefficient  except  for 
prescribed,  restricted,  and  highly  formalized  "planning". 

A planning  job  or  task  performed  by  an  individual  is  usually  restricted  to  an 
information  domain;  the  planning  task  deals  with  a restricted  set  of  variables 
in  terms  of  goals,  relevant  environmental  states,  and  relevant  resources.  Eco- 
nomic planners  deal  with  cost,  revenue,  and  profit  variables;  inventory  planners 
deal  with  goods  flow  and  storage  variables;  aircraft  mission  planners  deal  with 
the  variables  of  payload,  distance,  targets,  routes,  equipment,  personnel. 

There  are  short-term  planners  and  long-term  planners,  and  each  works  in  a some- 
what different  domain  of  variables  (including  goal  variables).  Any  planner's 
job  deals  with  a relatively  defined  and  restricted  context  of  information. 

The  planner,  like  any  other  decision  maker,  tends  to  learn  or  to  create  an  impli 
cit  structure  or  "model"  of  the  variables  associated  with  his  areas  of  responsi- 
bility. In  some  cases,  this  model  may  become  as  explicit  as  a checklist. 
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Anticipation  of  requirements  can  be  standardized  if,  in  every  planning  cycle, 
the  requirements  tend  to  be  about  the  same.  Allocation  of  resources  tends  to 
be  about  the  same  If,  on  every  planning  cycle,  the  pool  of  resources  is  the 
same,  and  all  priorities  are  about  the  same.  Under  such  conditions,  planning 
becomes  the  operation  of  a set  of  rules  and  formats. 

Since  planning  is  a special  form  of  decision  making,  the  reader  should  refer  to 
the  entire  subsection  on  Decide/Select.  The  material  in  that  discussion  is  not 
repeated  here. 

Behavior  Tendencies 


See  Behavior  Tendencies  described  under  Decide/Select. 

Task  Strategies 

The  Task  Strategies  cited  under  Decide/Select  should  be  referred  to,  along  with 
the  following. 

Critical  path  schematics.  The  concept  underlying  PERT  (Program  Evaluation  Re- 
view  Technique)  and  critical  path  methods  is  the  fundamental  structure  for  any 
kind  of  planning  operation.  In  summary,  this  consists  of  (a)  constructing  a 
scenario  of  the  major  points  along  the  route  towards  completion  of  an  objective, 
and  (b)  determining,  for  each  of  these  major  points,  necessity  for  desirable 
resources  for  proceeding  into  the  next  phase  of  activity  towards  the  objective. 

It  is,  in  effect,  a sequence  chart  of  parallel  and  converging  actions.  A major 
point  may  be  defined  as  a position  in  a path  where  a new  pattern  of  resources 
is  required,  or  some  condition  or  state  must  be  achieved. 

Consider  a plan  for  a business  trip  to  another  city.  Planning  is  required  for: 
getting  off  on  time,  transportation  to  the  airport,  an  airplane  ticket  to  the 
destination,  transportation  from  the  destination  airport  to  the  office  to  be 
visited,  arrangements  for  meeting  the  desired  people  there  at  an  expected  time, 
transportation  back  to  the  airport,  return  passage,  and  so  forth.  This  scenario 
suggests  the  elements  to  be  planned  for;  i.e.,  transportation  to  airport,  air- 
plane reservation,  and  so  forth. 

Naive  planners  expect  transportation  to  be  ready  when  called,  airplanes  to  be 
strictly  on  time,  hotel  reservations  always  to  be  in  order.  Where  success  is 
extremely  important,  good  planning  anticipates  contingencies  and  prepares  some 
fallback  for  them.  Planning  for  contingencies  inevitably  calls  for  redundant 
or  additional  resource(s)  to  be  available.  Judgment  (based  on  a combination 
of  knowledge  and  imagination)  estimates  the  chances  of  given  things  going  wrong, 
the  chances  of  an  impromptu  remedy  being  available,  and  balances  these  against 
the  cost  of  providing  a fallback  resource. 

One  class  of  resource  can  often  be  balanced  against  another.  If  you  are  concerned 
about  having  a reservation  for  a given  flight  at  10  o'clock  in  the  morning,  you 
might  (if  unethical)  arrange  booking  for  an  alternate  flight  at  about  10  o'clock 
to  the  same  destination  on  another  airline.  This  could  cost  dollars  or  reputa- 
tion. If  you  had  ample  time  resource  and  were  willing  to  wait  until  the  11 
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o'clock,  flight  in  case  the  one  booked  at  10  o'clock  did  not  work  out,  this  would 
be  a fallback. 


Strategic  planning  --  taking  into  account  contingencies  --  seeks  for  resources 
that  are  least  costly  in  terms  of  the  organization’s  resource  and  most  versa- 
tile in  terms  of  potential  mission  contingencies.  In  civilian  and  garrison 
operations,  money  is  a highly  versatile  resource,  but  it  has  little  value  when 
a carrier-based  aircraft  is  running  out  of  gas  one  hundred  miles  from  the  deck. 


The  tighter  the  plan  in  terms  of  permissible  resource  to  be  committed  (especially 
time),  the  more  vulnerable  is  success  to  the  occurrence  of  the  unexpected.  This 
is  why  a vast  amount  of  planning  went  into  thinking  through,  in  advance,  the 
variety  of  contingencies  that  could  occur  in  manned  space  missions.  Every  mission 
has  had  one  or  more  contingencies.  Incidentally,  one  should  not  forget  the  human 
as  a planned  resource  to  be  treated  as  a range  and  degree  of  competence  whose 
costs  include  selection,  training,  and  maintenance. 

In  summary,  a dimension  of  strategic  planning  consists  of  creating  a schematic 
or  scenario  of  the  flow  of  events  in  a mission  and  of  checkpoints  where  a new 
class  of  resources  is  required.  That  schematic  should  suggest  what  resources 
are  needed,  and  when,  and  should  also  point  to  the  possible  contingencies  re- 
quiring fallback  resource. 

Contingency  resources.  Here,  we  continue  a theme  from  the  previous  discussion. 

A statement  in  quantitative  terms  is  proposed  which  is  based  on  a variety  of 
experiences  and  recommendations  from  both  formal  and  informal  planners.  After 
careful  planning  and  resource  budgeting,  allow  approximately  20  percent  more  of 
the  budgeted  resource  for  contingencies,  if  the  mission  occurs  on  a well-known 
route  with  well-known  resources.  But,  if  the  route  Is  unfamiliar  or  uncertain, 
and/or  the  resource  capability  is  not  well  known  under  the  mission  conditions, 
allow  substantially  more  than  a 20  percent  reserve  in  budgeted  resource. 

These  may  be  a pseudo-quantification  except  where  dollars,  time,  or  materials  are 
concerned.  The  actual  values  may  be  challenged,  and  if  there  is  a better  empiri- 
cal background  for  selecting  a different  value  for  a class  of  planning  operations, 
the  better  value  should  be  used  as  a training  and  operating  guide. 

Training  Implications 

Few  duties  and  responsibilities  do  not  entail  some  planning.  Formal  curricula 
would  profitably  develop  conceptual  and  practice  content  in  the  art  and  disci- 
pline of  planning  in  the  context  of  the  given  duty  and  responsibility.  In 
daily  life,  at  work  or  in  private,  the  fact  is  that  few  individuals  seem  to  be 
very  good  planners.  Planning  is  not  a skill  that  is  necessarily  picked  up 
spontaneously  through  on-the-job  experiences. 

At  the  last,  training  might  provide  examples  of  structural  planning  formats  for 
various  relevant  job  contexts.  The  "format"  is  not  merely  a printed  form,  but 
also  Includes  some  exercises  in  constructive  thinking. 
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Holding  something  temporarily 

Rules  and  facilities  for  holding  messages  or 
parts  of  messages  in  temporary  storage  for  use 
at  later  times  during  a task  or  duty  cycle,  or 
for  combining  with  other  information  during 
the  cycle. 

The  above  is  a convenient  definition  for  short-term  memory,  perhaps  more  appro- 
priately called  "working  memory." 

In  general,  operations  are  equivalent  to  those  described  for  Recall  Enabling 
Information  (Long-Term  Memory). 

Examples 


t A typist  reading  a sentence  and  holding  It  in  mind  while  her  fingers 
key  the  symbols. 

t A computer  register  holding  data  presently  to  be  combined  with  other 
data  in  an  operation. 

• The  retention  of  symbols  or  messages  in  a buffering  device  for  trans- 
lating into  a different  transmittal  rate  or  frequency. 

• A decision  maker  holding  in  mind  the  objectives,  problem  information, 
and  choice-making  policies  in  the  course  of  reaching  a decision. 

• A troubleshooter  keeping  in  mind  the  tests  he  has  already  made  and 
his  Inferences  from  those  tests,  while  selecting  the  next  test. 

Factors 

t The  greater  the  number  of  input  channels,  variables,  codes,  and  chunks 
of  information  input  that  must  be  Integrated  over  time  to  reach  a de- 
cision or  select  a response  pattern,  the  larger  the  short-term  storage 
capability  required.  Human  short-term  memory  is  limited  (according  to 
a variety  of  factors)  but  can  be  functionally  increased  by  practice  of 
context  and  by  regularizing  or  formatting  the  input,  by  mnemonic  aids, 
and  by  map- like  job  aids. 

• The  greater  the  variations  among  message  sizes  and  message  rates  of 
transmitted  and  processing  data,  the  more  important  the  design  of 
short-term  storage  facilities  Is  to  the  efficiency  of  the  total  sys- 
tem. The  greater  the  kinds  and  ranges  of  random  variability  in 
messages,  message  sources.  Interruptions,  and  noise,  and  the  greater 
the  variety  and  complexity  of  response  pattern  alternatives,  the 
greater  the  burden  on  short-term  memory. 

• Information  elements  in  short- term  storage  must  be  functionally  address 
able  as  and  when  relevant  to  the  decision  to  be  reached  or  the  problem 
to  be  solved.  Some  address  codes,  or  mnemonics,  are  more  efficient 
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than  others  in  a given  system.  A given  "system"  in  this  context  is 
an  individual  who  is  likely  to  differ  from  others  in  his  mnemonic 
preferences. 


Discussion 


The  complexity  and  variety  of  short-term  memory  phenomena,  and  the  central  impor- 
tance of  this  function  to  tasks  that  have  cognitive  or  perceptual  content,  make 
short-term  memory  the  most  fruitful  of  all  human  functions  for  strategy  design. 

In  fact,  practically  every  function  in  the  present  set  has  direct  or  indirect 
bearing  on  short-term  memory,  and  many  of  the  strategies  recommended  for  other 
functions  might  as  readily  have  been  cited  here  under  short-term  memory. 

Individual  differences  in  cognitive  ability  should  nowhere  show  themselves  more 
significantly  than  in  short-term  memory,  except  perhaps  for  long-term  memory. 

This  would  seem  a truism.  Clearly,  the  complexity  and  scope  of  thinking  and 
problem  solving  seems  to  consist  largely  in  the  kind  and  amount  of  information 
an  individual  can  hold  in  mind  at  one  time.  The  ability  of  some  individuals  to 
hold  in  mind  long  strings  of  mathematical  code,  for  example,  while  others  can 
hold  in  mind  large  amounts  of  information  in  the  form  of  a map,  while  still 
others  can  hold  the  representation  of  an  intricate  model  or  processing  opera- 
tions in  mind  --  and  the  ability  in  each  case  to  "think"  with  that  content  -- 
clearly  implies  at  least  some  of  the  vast  differences  in  our  individual  mental 
capabilities. 

% 

We  have  individual  "preferences"  for  the  kind  of  code  or  cognitive  representa- 
tion of  data  presented  to  us  for  thinking  purposes  in  short-term  retention. 
Experienced  tutors  learn  that  one  way  of  representing  a concept  or  pattern  of 
relations  and  operations  to  a student  may  be  almost  completely  useless  to  his 
comprehension,  whereas  another  symbolic  structure  of  the  same  content  may  quickly 
communicate  the  ideas. 

Individual  differences  in  the  kinds  of  code  that  can  be  readily  apprehended  and 
retained  in  short-term  memory  partly  explain  the  emergence  of  different  patterns 
of  performance  ability  among  individuals  at  different  stages  of  learning.  At 
early  stages  of  learning,  the  ability  to  remember  and  translate  verbal  instruc- 
tions regarding  "what  to  do  next"  may  be  the  major  differentiator  among  rapid 
and  slow  learners.  But  at  higher  levels  of  task  learning,  when  these  self- 
instructing  mediators  tend  to  drop  out,  the  code  patterns  and  "conceptual  models" 
of  the  task  content  itself  dominate  performance  levels.  A person  who  is  poor 
at  remembering  strings  of  verbal  instructions  in  learning  how  to  go  about  reading 
a map,  may  be  very  good  indeed  at  remembering  the  map  content  itself,  once  he 
has  mastered  the  procedural  operations  for  inspecting  maps  for  a given  task  pur- 
pose. 

One  individual  may  be  quick  to  learn  the  verbal  self-instructional  content  of 
a set  of  manual  procedures,  but  be  relatively  poor  at  later  stages  of  learning 
that  depend  on  muscle  coordination  and  perhaps  on  the  ability  to  code  patterns 
of  perceptual  information  and  transform  it  "spontaneously"  (rather  than  cogni- 
tively) Into  patterns  of  muscle  activity. 
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The  Implications  for  instructional  technique  of  knowledge  about  individual  diff- 
erences in  short-term  "coding"  preferences  and  abilities  are  almost  certain  to 
be  more  important  to  training  technology  than  the  practice  of  letting  each  stu- 
dent practice  from  the  same  content  at  his  individual  learning  rate. 

The  content  of  short-term  is  not  necessarily  limited  to  what  the  operator  can 
tell  about  or  to  what  he  is  aware  of.  Short-term  memory  seems  to  have  sublimi- 
nal fringes.  One  may  be  aware  of  not  having  completed  some  task  action  without 
remembering  its  content.  While  reading  a page  of  text,  an  individual  may  be 
interrupted.  When  he  returns  to  reading  the  text,  his  eye  tends,  spontaneously, 
to  find  the  location  on  the  page  where  the  interruption  occurred.  He  spontane- 
ously begins  reading  where  he  stopped  at  the  interruption.  The  place  where  "the 
eye"  was  interrupted  must  have  been  stored  in  short-term  memory. 

Short-term  memory  may  well  be  the  basis  for  the  organizing  of  data  that  results 
in  the  selection  of  a "set  to  respond."  Data  organization  may  be  quite  complex, 
as  when  one  speaks  and  listens  in  French  to  the  dinner  companion  on  the  left 
and  In  Chinese  to  the  guest  on  the  right.  In  sight  reading  of  music  from  a 
score,  the  key  signature  indicates  the  pattern  of  notes  to  be  struck  when  a 
given  symbol  for  a note  is  shown  on  the  staff.  This  key  signature  invokes  a 
very  specific  repertoire  set.  But  sometimes  an  "accidental"  occurs  --a 
symbol  in  front  of  a note  that  countermands  the  signature.  The  key  signature 
may  indicate  that  a note  should  be  flatted,  which  the  accidental  symbol  says 
should  be  played  "natural."  The  accidental  value  for  the  note  applies  through- 
out the  measure,  and  the  musician  must  remember  to  play  the  same  note  with  the 
accidental  value  whenever  it  reappears  in  that  measure;  in  the  next  measure, 
the  accidental  value  no  longer  applies.  This  is  an  example  of  a set  within  a 
set  which  must  be  managed  within  short-term  memory. 

It  is  not  necessary  to  imply  that  the  code  to  an  entire  response  repertoire 
making  up  a "set  to  respond  to  a context"  must  be  held  in  short-term  memory 
work  space  --  if  indeed  there  is  such  a "space."  But  it  is  necessary  to  postu- 
late some  kind  of  code  representation  in  short-term  memory  which  selectively 
accesses  that  bounded  repertoire. 

In  short-term  memory  the  structure  for  a command  to  search  long-term  memory  is 
created.  We  are  now  speaking  of  a voluntary  search  into  deep  memory.  "How  in 
the  past  did  I remove  a gasket  from  an  engine  block  without  tearing  the  gasket?" 
Individuals  differ  in  the  format  and  code  that  is  congenial  in  long-term  memory, 
just  as  they  do  in  short-term  memory  --  and  probably  in  quite  parallel  ways. 

From  our  personal  history  of  successes  and  failures  in  memory  searches,  we  know 
that  starting  with  one  kind  of  picture  or  pattern  seems  to  have  better  chances 
of  uncovering  the  information  we  seek  than  do  other  patterns.  (The  subsection 
on  Recall  Enabling  Information  (Long-Term  Memory)  contains  an  extended  discussion 
of  this  issue.)  To  the  extent  that  education  and  training  are  not  concerned 
merely  with  getting  students  to  pass  tests,  they  could  be  directed  to  the  design 
of  what  logicians  call  "search  arguments"  in  the  psychological  sense.  The  con- 
struction of  a search  argument  is  the  equivalent  of  a strategy,  used  deliberately 
by  the  student  for  searching  "what  he  knows."  Unless  the  search  picture  (or 
argument)  can  effectively  be  held  in  short-term  memory  as  a single  piece  or 
pattern,  the  capacity  for  search  is  limited. 
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In  an  associative  system  (in  the  stimulus-stimulus  sense  rather  than  the  stimu- 
lus-response sense)  when  a "response"  is  made,  it  elicits  the  predisposition  to 
make  other  responses  with  which  it  has  associative  links.  As  these  responses 
occur  (implicitly  and  subl iminal ly  or  explicitly  with  awareness),  they  generate 
an  expanding  web  of  still  other  responses.  The  fact  of  the  occurrence  of  the 
response,  even  subliminally,  strengthens  the  predisposition  to  the  entire  re- 
sponse family  in  the  web.  It  is  like  initiating  a reverberation  into  a self- 
powered  network  with  an  internal  positive  feedback.  It  is  this  process  of  asso- 
ciative arousal  of  a set  --  a predisposition  to  respond  with  a large  vocabulary 
code  of  responses  --  that  takes  time.  This  way  of  conceptualizing  the  problem 
enables  a considerable  number  of  variables  and  hypotheses  to  be  deduced  about 
relative  lengths  of  time  that  might  be  required  to  induce  various  kinds  of  sets. 
In  forming  these  hypotheses,  one  should  remember  that  in  an  associative  system 
the  information  "content"  and  the  "code"  in  which  it  is  stored  tend  to  be  iden- 
tical to  each  other,  at  least  from  an  operational  standpoint. 

These  general  comments  about  short-term  memory  may  serve  less  as  direct  tutorial 
guidance  for  procedures  that  generate  short-term  memory  strategies,  than  to  in- 
spire confidence  that  short-term  memory  phenomena  present  rich  opportunity  for 
the  development  and  application  of  behavioral  strategies,  both  in  the  shape  of 
human  engineering  job  supports  and  in  instructional  technique. 

Behavioral  Tendencies 


With  continued  practice  of  a task,  and  the  processing  of  information  associated 
with  that  task,  a number  of  tendencies  can  be  expected. 

Reduction  of  content.  The  amount  of  information  that  passes  through  cognitive 
awareness  tends  to  be  reduced  to  some  level  established  by  the  monitoring  prac- 
tices of  the  operator  for  feedback  sampling.  That  is,  progressively  less  "mean- 
ingfulness" is  projected  by  the  operator  from  long-term  memory  onto  standard 
data  than  is  perceived  and  processed.  This  can  be  tested  roughly  by  determining 
how  many  events  the  operator  is  able  to  report  that  were  unique  to  a given  task- 
cycle  or  set  of  task-cycles. 

Standardized  formats.  More  or  less  standardized  formats  or  matrices  for  the 
short-term  retention  of  information  develop.  An  example  of  some  of  these  for- 
mats for  tasks  with  complex  non- repetitive  dependencies  between  input  informa- 
tion and  output  response  is  that  of  imagery  about  objects  and  processes.  Imagery 
is  one  major  kind  of  stable  memory  reference.  The  implicit  complex  image  has 
components  updated  by  samples  of  input.  (Some  innate  mechanism  seems  capable 
of  regenerating  this  image  and  selectively  updating  it.)  An  example  is  the 
pilot's  image  of  the  attitude  of  an  aircraft;  understanding  this  property  enables 
one  to  predict  when  the  "outside-in"  representation  is  better  than  "inside-out" 
and  vice  versa.  Another  example  is  that  of  the  schematic  image  a mechanic  has 
of  an  engine  when  he  is  making  inferences  from  diagnostic  tests. 

Standardized  formats  for  holding  information  in  short-term  memory  are  identical 
to  those  posited  for  decision  making  in  a later  subsection. 

Sampling  operations.  There  are  interactions  between  the  perceptual  sampling 
process  ana  short-term  memory  capabilities  with  respect  to  size  of  clump  of 
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information  sampled  at  one  time.  Too  large  a clump  for  a given  level  of  skill 
results  in  a reduction  of  performance  effectiveness.  Size  of  clump  is  also  in- 
fluenced by  time  pressures  from  competing  demands  for  attention,  and  by  its  con- 
formity to  individual  level  of  subjective  comfort.  Unless  modified  by  deliber- 
ate operator  intent,  or  by  specialized  training,  the  size  of  clump  stabilizes 
at  some  size  less  than  that  of  the  capability  of  the  operator,  and  quite  poss- 
ibly less  than  optimum  for  the  task.  This  tends  to  happen  among  many  readers 
of  straight  text  — at  least  according  to  the  proponents  of  high  speed  reading 
for  the  general  population. 

Task  Strategies 

Anticipatory  sampling.  Where  the  task  imposes  time  dependent  relationships  be- 
tween  input  and  output,  one  objective  is  to  sample  from  information  sufficiently 
in  advance  of  the  time  that  effector  response  needs  to  be  made  to  that  informa- 
tion, so  that  the  effector  response  can  become  organized.  Sampling  ahead  can 
provide  time  for  checking  supplementary  channels  of  information.  Most  import- 
ant, the  sampling  ahead  of  effector  response  enables  maintenance  of  a relatively 
even  rate  of  effector  processing  flow  — and  avoids  working  by  fits  and  starts. 
The  training  strategy  is  to  force  the  operator  to  scan  ahead  of  his  effector 
activity,  after  some  degree  of  automaticity  and  organization  of  effector  acti- 
vities has  been  acquired,  or  more  or  less  coincidental  with  that  stage  of  learn- 
ing. 

Conceptual  structure.  Where  relatively  large  amounts  of  variable  information 
must  De  held  in  short-term  storage,  suitable  retention  of  that  material  is  re- 
quired in  a form  that  may  be  independent  of  the  sequence  in  which  it  is  converted 
into  effector  response.  The  task  strategy  is  to  develop  a conceptual  "model 
which  may  include  the  support  of  imagery  (of  various  kinds  including  visual, 
auditory,  kinesthetic)  which  serves  as  an  updatable  reference.  Where  possible, 
the  image  should  partake  of  the  goal  image  content  and  should  be  organized  with 
similar  content.  The  image  may  be  supported  by  a matrix  of  verbally  specifi- 
able variables  of  relevance  to  sampling  and  to  effector  execution.  Ideally, 
the  reference  image  is  comprehensive  enough  to  provide  a single  reference  for 
all,  or  most,  of  the  most  critical  channels  of  information.  A mental  picture 
of  the  attitude  of  an  airplane  in  reference  to  the  horizon  embraces  the  vari- 
ables of  roll,  pitch,  and  yaw;  information  about  each  of  these  variables  may 
be  transmitted  through  a separate  display  channel. 

Short  and  long-term  memory  code.  The  content  and  code  of  short-term  memory 
should  be  consistent  with  the  code  and  "indexing”  of  information  in  long-term 
memory,  so  that  retrieval  of  information  or  "programs"  from  long-term  memory 
is  fast  and  reliable.  This  is  a function  of  training. 

Stereotypical  performance.  The  tradeoff  against  formatted  short-term  memory 
may  be  stereotypical  projections,  and  the  belief  that  old  regenerated  data  are 
really  new  data  as  presented  from  input  sources.  Good  short-term  memory  is  de- 
pendent upon  attentive  observation.  These  issues  were  discussed  in  our  treat- 
ment of  Detect,  Identify,  and  Interpret.  Obviously,  all  of  these  processes 
interact  both  behavioral ly  and  in  their  consequences  in  performance.  Redun- 
dant sampling  of  the  situation  may  mitigate  against  stereotyping  that  is  mal- 
adaptive in  performance.  Fundamentally,  stereotypical  performance  is  adaptive 
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from  an  internal,  behavioral  standpoint  — it  saves  effort  and  time  in  perceptual 
discriminations,  choice  making,  and  effector  response. 

Impulse  control.  Short-term  memory  should  also  guard  against  localized  impulsive 
benavior.  The  impulse  to  drop  an  uncomfortably  warm  object  when  it  is  grasped, 
should  be  counterbalanced  when  the  object  is  an  expensive  vacuum  tube.  The 
impulse  may  be  to  swerve  to  the  left  when  the  car  ahead  brakes  suddenly,  but 
short-term  memory  that  recalls  another  car  approaching  fast  on  the  left  may 
block  the  impulse  out  of  the  dominance  in  the  response  hierarchy. 

Filtering  in  short-term  memory.  Since  short-term  memory  is  (or  consists  of) 
holding  the  cognitive  content  that  is  at  least  one  level  of  associative  content 
which  is  "set  to  respond,"  memory  is  also  the  basis  for  selective  response  to 
information  "relevant"  to  goal  objectives  or  operator  "purpose."  Note  that  cri- 
teria of  relevance  must  be  applied  not  only  to  environmental  input,  but  also  to 
internal  content  from  the  operator's  long-term  memory.  The  clarity  with  which 
a conceptual  model  is  established  determines  the  precision  with  which  relevant 
information  is  selected  and  information  "non-relevant"  to  the  model  is  filtered 
out.  The  mechanism  or  principle  of  action  for  this  filtering  (aside  from  the 
metaphor)  is  admittedly  obscure. 

Goal  image  and  conceptual  guidance.  Unless  the  task  has  been  procedural ized, 
the  image  of  goal  objectives  must  be  active  in  the  guidance  of  perceptual - 
cognitive-effector  behavior,  and  establish  both  relevance  and  priorities  of 
information  processing  content.  The  goal  image  must  also  be  active  in  short- 
term memory  when  there  is  any  kind  of  improvisational  behavior  by  the  operator. 

It  is  possible,  but  not  necessary,  that  the  operator's  concept  of  a model  of 
"what  is  happening"  in  the  field  in  which  he  is  operating  overlap  with  his  con- 
cept or  model  of  the  goal  state  he  is  attempting  to  create  or  maintain. 

A performance  strategy  for  short-term  memory:  Ideally,  the  conceptual  model 

the  operator  should  learn  and  exercise  representing  his  'picture  of  what  ij^ 
going  on"  in  his  field  of  operation  should  be  consistent  with,  and  at  least 
partly  coincidental  with,  his  image  of  a model  of  what  should  be  going  on 
with  respect  to  the  achievement  of  goal  states.  For  examp leT  the  driver  should 
have  a goal  image  of  what  his  driving  "slot"  should  be  for  safety,  comfort, 
and  so  on.  The  way  this  picture  or  image  of  the  idealized  slot  appears  to  him 
conceptually  should  enable  a kind  of  image  "overlay"  of  what  he  actually  ob- 
serves to  be  fact  during  his  psychological  present  while  he  is  driving.  His 
actual  driving  behavior  thus  becomes  a compromise  between  the  exigencies  of 
the  environment,  and  the  deviations  from  the  pattern  or  image  of  the  idealized" 
state  compared  with  that  of  the  actual  state.  It  is  not  necessary  to  stipulate 
the  capacity  of  the  operator  to  modify  the  image  of  idealized  state  to  take 
into  account  various  constraints  in  the  environment  and  in  himself.  For  example, 
when  he  is  searching  for  route  indicators  (such  as  street  signs)  in  a downtown 
area,  his  idealized  image  should  have  larger  buffer  zones  in  terms  of  distance 
of  cars  ahead  and  especially  behind  his  car,  than  in  the  absence  of  this  con- 
tingency. If  he  is  aware  of  drowsiness  and  lack  of  alertness,  he  may  similarly 
expand  his  conceptual  image  of  margin  of  safety  or  buffer  zone  for  permissible 
action  to  occur. 
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Performance  strategy  also  demands,  ideally,  that  the  operator's  active  conceptual 
model  be  inclusive  of  all  the  major  task  variables  about  which  status  data  should 
be  obtained  and  retained  in  short-term  memory.  This  point  has  been  underlined  in 
those  topics  which  have  emphasized  the  formatting  of  task  information  in  task  be- 
havior. 
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Holding  something  permanently 

Rules  and  operations  for  holding  messages  and 
message  content  for  retrieval  in  terms  of  where, 
how,  and  in  what  form  (code).  These  include 
rules  for  filing  and  retrieval  search.  The 
contents  of  storage  are  information  about  data 
and  information  about  procedures. 

In  the  human,  the  distinction  between  information  about  data  and  about  task  pro- 
cedures may  be  arbitrary.  The  major  emphasis  here  will  be  on  that  form  of  mem- 
ory that  is  available  to  thought  and  awareness. 

Deep  and  Surface  Memory 

A distinction  between  deep  and  surface  memory  is  an  artificial  one,  made  to  em- 
phasize a point.  If  you  are  asked  your  current  telephone  number,  or  for  a gen- 
eral description  of  your  child,  your  answer  is  likely  to  be  direct  and  immediate. 
This  is  what  may  be  called  surface  memory.  If  you  have  moved  as  often  as  the 
average  man  and  are  asked  your  telephone  number  of  ten  years  ago,  or  if  you  are 
asked  to  describe  your  12-year  old  child  as  an  infant,  you  are  likely  to  pause, 
reflect,  and  search  through  various  trains  of  ideas  and  associated  contexts  in 
order  to  retrieve  the  information  necessary  to  respond.  An  attempt  is  made  to 
elicit  the  information  by  building  up  an  associative  context  that  serves  as  a 
key  to  the  target  information.  This  is  probing  deep  memory. 

The  distinction  between  surface  and  deep  memory  is  mostly  a matter  of  degree  and 
of  the  availability  of  context  keys  for  search.  The  important  point  is  that  on 
some  occasions,  as  in  the  handling  of  rare  contingencies,  the  operator  may  not 
have  the  requisite  information  at  the  "surface  level",  but  may  have  it  at  deeper 
levels  where  memory  search  strategies  have  practical  meaning. 

Major  Tradeoffs 

Storage  in  form  of  rules  or  principles  versus  specific  content.  Task  information 
may  be  stored  in  the  form  of  rules  and  principles  which  require  logical  operations 
in  order  to  deduce  or  induce  the  specific  "answer"  to  an  immediate  situation;  or, 
the  equivalent  task  information  may  be  stored  as  specific  content  to  be  elicite? 
by  specific  situations.  Thus,  a student  may  learn  the  rules  for  calculating  the 
square  root  of  a number,  or  he  may  memorize  a square  root  table.  Storage  in  the 
form  of  rule,  procedure,  or  principle,  may  use  less  "memory  space"  but  requires 
computational  time  and  is  subject  to  computational  error.  It  is,  however,  capable 
of  extensive  extrapolation  to  situations  not  encountered  in  direct  experience. 

Memorization  of  tables  of  data  and  rote  procedures,  on  the  other  hand,  requires 
more  memory  space  --  it  takes  longer  to  learn  --  but  requires  less  time  to  pro- 
duce — recall  --  and  is  free  of  computational  error.  It  is  subject  to  memory 
error  and  the  interference  effects  characteristic  of  rote  materials  (i.e.,  ma- 
terials having  little  associative  context). 

In  the  case  of  arithmetic,  educational  practice  combines  both  storage  methods. 

We  memorize  the  multiplication  table  of  numbers  from  1 to  12  (very  likely  with 
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the  exception  of  multiplication  by  10)  and  learn  rules  for  deriving  products  of 
larger  numbers. 

Material  that  is  used  very  frequently  should  be  stored  for  rapid  and  direct  ac- 
cess; material  used  less  frequently  or  less  repetitively  may  more  efficiently 
be  stored  as  a rule  or  principle  for  application  as  needed. 

A liability  in  teaching  and  performing  tasks  by  "principle"  is  that  the  opera- 
tor may  not  be  able  to  recall  the  correct  identification  of  the  principle  when 
a situation  calls  for  its  application. 

If  the  information  system  has  a fast  processor  and  a small  rote  memory  capa- 
city, computed  answers  are  obviously  the  preferred  choice.  If  the  processor 
(perhaps  some  kind  of  "intelligence")  is  slow  or  restricted  by  small  working 
memory,  dependence  must  be  made  on  the  basis  of  rote  memory  and  "table  lookup." 

Parsimony  versus  richness  (redundancy)  in  storage  content.  Here  we  have  a sec- 
ond  major  tradeoff.  Thus,  a procedure  may  be  learned  by  direct  repetition  in 
one  hundred  trials  to  a high  degree  of  reliability,  perhaps  automaticity.  An- 
other training  regimen  may  consist  not  only  of  practice  in  the  procedure  in  a 
rote  fashion,  but  also  include  conceptual  content  in  the  sense  of  making  the 
procedure  "meaningful,"  perhaps  by  indicating  the  effects  of  each  part  of  the 
procedure  and  of  the  procedure  in  its  entirety. 

Mnemonic  richness,  or  associative  context  --  or  meaningful  ness  --  may  support 
retention  and  recall  of  the  operative  task  information.  Its  effectiveness  de- 
pends upon  the  art  by  which  the  mnemonics  are  designed  and  associated  by  the 
learner  with  the  task  content.  A mnemonic  may  be  psychologically  irrelevant, 
thereby  imposing  an  information  handling  burden  rather  than  serving  as  an  aid. 
The  proportion  of  learning  time  devoted  to  direct  practice  and  that  given  to 
mnemonic  aids  may  also  require  intuitive  judgment,  often  on  the  basis  of  diff- 
erences among  individual  learners.  (We  assume  that  in  real  work  situations 
the  luxury  of  extensive  experimental  research  is  prohibited.) 

Recall  by  actual  temporal  association  versus  association  in  thought.  The  pri - 
mary  basis  for  the  association  of  cognitive  content  in  memory  is  temporal  con- 
tiguity in  the  experience  of  the  content.  This  contiguity  may  have  occurred 
in  the  original  experience.  For  example,  meeting  a man  named  J.  J.  Roe,  who 
was  wearing  the  uniform  of  a military  officer,  on  the  steps  of  the  Lincoln 
Memorial  — any  two  of  the  three  sets  of  cues  might  be  sufficient  to  elicit 
many  of  the  undescribed  cues  in  that  experience.  The  greater  the  number  of 
cues  present  (spatially  and  temporally  proximate  in  the  original  experience), 
the  greater  the  probability  of  recall  (or  identification  or  reproduction)  of 
the  remaining  cues  in  the  experienced  situation.  The  greater  the  number  of 
pieces  that  make  up  an  experienced  pattern  presented  in  a retention  test,  the 
greater  the  chances  that  the  remainder  of  the  pattern  can  be  filled  in  by  re- 
call. But,  mental  rehearsal  with  various  cognitive  contexts  established 
another  temporal  contiguity  of  content  in  memory.  When  thoughts  occur  toget- 
her,  the  future  occurrence  of  one  of  these  thoughts  tends  to  elicit  the  other. 

These  assertions  about  recall  probability  assume  equality  of  other  influences  on 
retention,  such  as  interference  effects,  generalization,  and  stereotype. 
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The  following  principles  include  operations  for  cominiting  information  to  long- 
term storage  as  well  as  operations  for  retrieving  content  from  long-term  stor- 
age. Clearly,  some  acknowledgment  of  individual  differences  must  qualify  any 
generalizations  about  memory  processes.  Different  individuals  undoubtedly  have 
different  "preferred"  mnemonic  codes  --  verbal,  visual,  auditory,  kinesthetic  — 
for  storage  and  retrieval.  And,  any  list  of  classifications  is  probably  too 
simple  for  guidance. 

Similarity  of  learning  and  recall  contexts.  It  is  axiomatic  that  recognition, 
recall,  or  reproduction  from  memory  is  facilitated  to  the  extent  that  the  learn- 
ing conditions  and  the  conditions  in  the  retention  test  are  similar.  Similarity 
is  a matter  of  degree.  Psychological  or  implicit  similarity  is  more  important 
than  objective  similarity.  That  is,  the  contents  of  attention  at  the  time  of 
learning  are  of  key  importance  to  the  formation  of  associative  patterns.  Thus, 
what  is  perceived  and  conceptualized  in  the  course  of  attempting  to  learn  Task 
A tends  to  be  recalled  in  the  process  of  attempting  to  perform  Task  A.  Whatever 
was  not  perceived  while  learning  or  performing  Task  A will  be  relatively  insig- 
nificant for  the  recall  of  cognitive  or  perceptual -motor  content. 

The  matter  can  be  stated  another  way.  What  is  learned  in  the  real  context  of 
performing  Task  A is  perceived  and  thought  about  in  the  same  terms  as  a later 
recurrence  of  the  Task  A situation  presents.  No  change  in  coding  of  the  content 
is  required  (as  is  needed  when  one  learns  about  Task  A from  a textbook  and  must 
somehow  "translate"  the  verbal  content  into  the  actual  task  situation).  There 
is  a corollary  to  this  idea  which  is  a commonplace  recall  strategy.  If  an  oper- 
ator is  attempting  to  recall  a fact  or  procedure  that  he  once  learned  while  per- 
forming Task  A but  is  not  now  performing  in  the  Task  A environment,  he  should 
attempt  to  picture  himself  performing  Task  A. 

It  is  possible  to  learn  to  perform  a task  so  automatically  that  one  cannot  ver- 
bally describe  its  components.  Tying  one's  shoelaces  is  an  example  of  this  sort. 
Ask  most  any  highly  skilled  typist  where  the  letter  "f"  is  located  on  the  key- 
board, and  she  won't  be  able  to  tell  you  until  after  she  has  twitched  her  left 
forefinger.  This  example  suggests  a mnemonic  strategy  for  recall  of  information 
under  such  circumstances. 

Some  operational  specifics  about  learning  and  recall  similarities  are  discussed 
in  an  earlier  subsection  on  the  Code  function. 

Intent  to  learn.  Deliberate  intent  to  remember  what  is  being  experienced  in- 
creases the  probability  of  its  recall,  even  under  varying  circumstances  of  re- 
tention testing. 

Cl ari ty  of  content.  The  clearer  the  cognition,  the  better  it  will  be  remembered. 
An  immediate  test  of  clarity  of  cognitive  content  is  the  ability  to  paraphrase 
the  content.  If  cognitive  content  is  intended  to  support  the  recall  or  repro- 
duction of  procedures,  or  to  provide  a matrix  for  productive  thinking,  it  must 
be  grasped  in  the  active,  manipulative  sense  rather  than  merely  in  a passive  way. 
For  example,  if  an  electronics  technician  must  interpret  and  hold  in  mind  test 
data  relating  to  Ohm's  Law,  it  is  inadequate  for  him  merely  to  memorize  the 
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symbols  "E  = IR."  Some  mnemonic  image  for  the  “meaning"  of  E , I,  and  R not  only 
aids  retention  of  these  relationships  in  the  abstract  sense,  but  also  enables 
him  to  make  tests  and  hold  the  data  in  mind  more  reliably  in  terms  of  a concrete 
analogy.  The  analogy  can  assist  in  mental  manipulation  of  the  terms  in  the  equa- 
tion while  still  attaching  a picture  of  private  meaning  to  the  various  patterns 
in  which  the  relationship  can  be  expressed.  This  sense  of  concept  clarity  may 
be  irrelevant  to  classroom  tests  of  a verbal  kind;  it  is  certain  to  be  relevant 
to  actual  task  situations  where  behavior  is  guided  and  interpreted  — and/or 
where  the  recall  of  a set  of  procedures  is  to  be  aided  — by  the  concept  and 
principle. 

As  another  example,  consider  teaching  a novice  what  to  do  in  the  face  of  an  im- 
pending aircraft  stall.  As  a support  to  the  proper  procedure  (whose  execution 
is  against  a "natural"  tendency),  the  student  is  also  taught  the  aerodynamic 
concept  of  the  relationship  of  critical  velocity  and  lift.  The  student  forms  a 
mental  picture  of  this  principle  so  that  his  image  of  it  and  the  verbal  state- 
ment of  the  procedure  are  mutually  supportive.  If  the  image  is  clear,  when 
stall  speed  is  reached  he  will  properly  push  forward  on  the  stick  in  order  to 
gain  velocity  with  the  help  of  gravity  and  thus  get  help  in  overcoming  stall. 

Even  though  the  principle  is  well  rehearsed  in  non-stressful  circumstances,  if 
the  principle  is  not  sufficiently  clear,  it  may  interfere  with  proper  recall 
and  execution  of  the  correct  procedure. 

Stored  answers  versus  calculated  answers.  A stored  answer  is  a response  directly 
available  in  memory  for  use  upon  situational  demand.  A calculated  answer,  in  the 
present  context,  is  one  which  must  be  obtained  by  a procedure;  the  procedure  may 
be  logical,  arithmetic  or  mathematical,  or  it  may  be  the  result  of  a search  oper- 
ation, such  as  looking  for  an  answer  in  a reference  manual.  Psychologically,  the 
application  of  a procedure  uses  temporary  cognitive  capacity  as  well  as  time,  and 
may  interfere  with  the  operator's  train  of  thought.  For  many  tasks,  a small  num- 
ber of  "answers"  is  applicable  to  a very  large  proportion  of  situations.  With 
extended  practice  on  the  job,  frequently  used  answers  are  remembered  so  that  they 
no  longer  require  procedural  or  "figuring-out"  derivation.  This  process  may  be 
shortened  by  spending  some  time  in  rote  memorization  of  high  frequency  responses 
that  are  known,  if  adequate  accuracy  checks  are  available.  An  example  is  the 
new  stock  clerk  who  may  find  a requisitioned  part  by  consulting  a locator  refer- 
ence or  by  going  directly  to  the  shelf  where  the  object  is  stored.  (Note  that 
an  error  check  is  built  into  the  operation  of  the  shelf  if  the  object  is  cor- 
rectly labeled.) 

Redundancy  of  task  procedures  information.  In  order  to  maintain  reliability  in 
retention  of  infrequently  performed  procedures,  the  procedural  content  should  be 
learned  at  more  than  one  level  or  code.  This  Is  a key  concept  relating  to  the 
operational  effectiveness  of  human  memory.  The  perceptual -motor , verbal  proce- 
dures, and  "understanding  how  it  works"  are  examples  of  different  retention 
codes.  The  recall  of  verbal  procedures  enables  an  operator  to  perform  a task 
with  self-instructions,  although  perhaps  rather  awkwardly  compared  with  percep- 
tual-motor (automatized)  performance.  A conceptual  picture  of  the  processes 
that  are  interpreted  and  controlled  by  task  behavior  may  enable  the  operator 
to  deduce  procedural  requirements  (situation-response  patterns).  This  mode 
may  be  even  slower  and  more  awkward  than  performing  from  a well -rehearsed  set 
of  self-initiated  instructions.  Obviously,  if  the  latter  mode  is  to  be 
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effective,  it  must  on  occasion  be  rehearsed.  Rehearsal  must  be  independent  of 
the  cies  that  can  be  generated  by  perceptual -motor  performance,  or  else  opera- 
tional justification  for  the  redundant  habits  does  not  exist. 

The  earlier  example  of  the  novice  pilot  in  an  aircraft  about  to  stall  is  rele- 
vant here.  The  pilot  may  have  acquired  perceptual -motor  reflexes  that  are  pro- 
perly responsive  to  the  inminent  stall.  In  the  absence  of  these,  he  may  remem- 
ber what  his  instructor  told  him  to  do  in  such  circumstances.  Failing  to  re- 
member his  instructor's  advice,  he  may  have  conceptualized  the  aerodynamic  sit- 
uation, and  deduce  from  it  the  appropriate  response(s). 

There  are  many  situations  in  v/hich  planned  redundancy  of  procedural  information 
is  highly  desirable.  Emergencies  which  may  happen  rarely,  thus  affording  little 
direct  practice,  are  one  such  type.  Practice  Is  likely  to  be  in  a simulator 
rather  than  in  circumstances  of  operational  stress.  Another  candidate  for  pro- 
cedural redundancy,  whatever  the  amount  of  perceptual -motor  practice,  is  a con- 
dition of  operational  stress  --  especially  where  there  are  internal  tendencies 
leaning  toward  erroneous  response.  Furthermore,  even  an  automatized  perceptual - 
motor  series  of  responses  can  be  disrupted  and  leave  an  operator  helpless,  like 
the  piano  student  whose  "finger  memory"  cannot  overcome  an  inadvertent  error, 
bypass  it,  and  continue.  Where  there  may  be  interference  effects  among  a large 
number  of  fairly  similar  procedures  which  the  operator  must  remember,  an  enhance- 
ment of  appropriate  differences  can  be  gained  by  verbal  and  conceptual  redun- 
dancies. On  occasion,  some  adaptive  flexibility  (an  exercise  of  "judgment") 
may  be  required  of  a procedural  performance  and  effective  response  may  depend 
on  a cognitive  reference  for  the  mechanics  of  the  procedure. 

The  training  implication  here  is  that  the  several  levels  or  encodings  of  proce- 
dural information  must  be  factually  consistent  with  and  operationally  parallel 
to  each  other.  Furthermore,  the  operator  must  not  be  required  to  sift  and  re- 
arrange a melange  of  concepts,  some  relevant  and  some  irrelevant  to  demands  of 
the  operational  situation.  In  other  words,  any_  "theory  of  operation"  will  not 
do.  The  conceptual  content  must  be  carefully  selected  for  relevance,  and  learned 
Tn  the  general  context  of  procedures  with  which  it  should  be  associated,  and 
also  psychologically  contiguous  to  the  practice  of  those  procedures.  Only  by 
these  means  can  functional  redundancy  of  procedural  information  in  memory  be 
assured. 

Accessing  memory  with  compatible  code.  If  you  briefly  see  a landmark  along  a 
roadway,  you  may  not  recognize  a verbal  description  of  it  but  you  may  recognize 
the  object  if  you  see  it  again,  or  if  presented  with  a photograph  or  perhaps 
even  a sketch  of  it.  If  a maintenance  technician  has  serviced  a number  of  pieces 
of  electronic  equipment  with  the  help  of  a particular  set  of  functional  schematic 
diagrams,  he  is  more  likely  to  remember  the  procedures  and  idiosyncrasies  of 
each  of  those  equipment  items  if  given  the  same  diagrams  to  work  with  at  a later 
date  than  if  given  a set  of  circuit  schematics,  even  if  these  contain  informa- 
tion more  or  less  equivalent  from  an  abstract  or  logical  point  of  view.  Even 
the  content  of  a photograph  or  map,  if  inverted  from  its  normal  orientation  to 
the  user,  is  less  reliably  recognized,  identified,  or  discriminated  than  when 
the  orientation  is  the  same  as  that  commonly  experienced  in  using  the  photo  or 
map. 
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These  are  examples  of  code  incompatibility  (in  the  sense  intended  here)  with 
the  content  of  memory.  These  incompatibilities,  in  respective  degree,  impose 
liabilities  not  only  upon  recall  of  the  content  presented,  but  often  more  sig- 
nificantly to  the  factual  and  procedural  context  associated  with  them  --  con- 
text which  may  be  relevant  to  task  performance.  Code  translation  uses  up  cog- 
nitive capacity,  as  any  student  with  bare  mastery  of  a foreign  language  realizes 
when  he  is  attempting  to  translate  a conversation  while  it  is  going  on.  Cog- 
nitive capacity  may  be  so  overextended  that  the  operator  is  not  ev6n  able  to 
retain  the  meaningful  context  of  the  messages  which  would  enable  him  to  recon- 
struct information  not  directly  apprehended.  He  may  not  recognize  that  a story  is 
being  told  that  he  knows  well  in  his  native  tongue. 

The  particular  "set"  to  respond  to  a given  context  of  information  may  be  thought 
of  in  functional  terms  as  part  of  the  code  for  appropriate  recall  of  a class  of 
data,  or  may  at  least  be  thought  of  as  a basis  for  selecting  a code  set.  (The 
pun  here  is  deliberate.)  Reading  nearly  illegible  handwriting  --  at  least  for 
some  readers  --  may  be  helped  by  knowledge  that  the  writer,  although  writing  in 
English,  was  educated  in  German  schools.  It  may  help  to  hear  that  a series  of 
lines  of  symbols  is  FORTRAN  code  for  a computer  control  program.  The  utility 
of  cues  that  establish  a set  depend,  of  course,  on  the  operator's  selective  pre- 
dispositions to  respond  to  such  cues. 

The  practical  significance  in  strategies  of  entering  long-term  memory  content 
for  recall  and  reproduction  purposes  is  enormous.  What  may  seem  to  be  logical 
equivalent  of  information  in  two  codes  may  be  psychologically  irrelevant  for 
recall  purposes. 

The  general  strategy  is  simple:  Enter  information  intended  for  long-term  recall 

in  the  same  form,  structure,  representation,  and  symbol  sets  that  will  occur  un- 
der operational  conditions  for  recall.  Thus,  If  a maintenance  technician  is 
tasked  with  repair  of  various  devices,  his  learning  of  the  procedures  for  those 
devices  should  be  from  the  same  manuals  and  job  aids  he  will  use  as  references 
during  actual  task  performance.  The  cues  given  him  for  introducing  a task  should 
be  similar  to  cues  given  him  in  the  operations  which  demand  the  selection  of  a 
task  set  preparatory  to  task  performance. 

Practical  exigencies  may  on  occasion  demand  that  the  operator  translate  informa- 
tion stored  in  one  code  into  another  code.  The  liabilities  in  doing  so  --  with- 
out extended  practice,  which  is  equivalent  to  learning  the  substitute  code  -- 
may  be  severe. 

Although  analogies  between  humans  and  machines  should  always  be  suspect,  it  is 
obliquely  pertinent  to  note  that  the  programming  of  a computer  interpreter  from 
one  computer  language  to  another  (e.g.,  from  FORTRAN  to  APL)  requires  a mammoth 
technical  and  financial  commitment. 

Assessment  of  the  reliability  of  memory  for  task  events  occurring  in  a sequence 
can  be  based  on  the  operator's  ability  to  anticipate  what  will  happen  next. 

This  means  that  the  operator  is  not  entirely  dependent  on  the  cues  immediately 
presented  to  him  for  the  recall  of  the  appropriate  action,  nor  is  he  entirely 
dependent  on  sensori -motor  memory. 
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If  the  operator  has  conceptual  mastery  of  a subject  matter  independent  of  any 
specific  task  actions,  then  a different  training  strategy  is  indicated  for 
reliability  of  recall.  The  equivalent  information  should  be  coded  in  a variety 
of  forms  during  training  — pictorial,  schematic,  verbal,  etc.  — in  relation 
to  many  contexts,  and  the  operator  should  be  required  to  abstract  the  relevant 
concepts  from  each  form  of  presentation.  Multiple  code  mastery  lifts  thq  oper- 
ator from  the  tyranny  of  any  specific  set(s)  of  cues,  and  enables  him  to  impro- 
vise responses  appropriate  to  the  concept  despite  the  coding  of  the  data  in  the 
situation. 

Mnemonic  reference  codes.  The  hierarchical  arrangement  of  subject  matter  titles 
in  a textbook  is  an  example  of  a reference  code  to  a body  of  subject  matter. 

If  a student  were  to  attempt  to  efficiently  memorize  and  efficiently  recall 
the  material  in  his  textbook,  after  a first  pass  of  study  of  its  contents,  he 
would  memorize  the  chapter  titles,  then  the  main  topic  titles  within  each  chap- 
ter, and  perhaps  then  the  subject  headings  within  each  topic  title.  To  the 
extent  that  he  could  project  "meaningfulness"  into  this  hierarchic  pattern  of 
relationships,  the  learning  and  remembering  would  be  more  efficient  and  effec- 
tive. A relatively  small  amount  of  memorizing  could  serve  to  reinstate,  in 
contextual  as  well  as  specific  associational  terms,  a large  amount  of  subject 
matter.  This  advice,  commonplace  in  study  procedures,  has  practical  implica- 
tions for  storage  and  recall.  Unfortunately,  this  excellent  counsel  seems  rarely 
to  be  practiced,  perhaps  largely  because  students  study  to  pass  tests  and  tests 
are  often  directed  to  specifics  of  subject  matter  rather  than  to  conceptual  or- 
gani zation. 

Many  jobs  in  real  life  require  knowledge  of  a large  collection  of  facts.  Ran- 
domly ordered,  these  facts  might  take  weeks,  months,  or  years  to  learn  for  re- 
call. An  example  might  consist  of  quartermaster  work  where  even  a segment, 
such  as  military  clothing,  may  consist  of  thousands  of  individual  items.  If 
direct  recall  is  desirable  in  many  transactions,  these  items  would  be  appropri- 
ately grouped  into  "logical"  classes  and  subclasses  with  class  and  subclass 
names.  A well-designed  taxonomy  can  be  a prodiguous  aid  both  in  learning  and 
to  recall.  The  fact  that  it  is  easier  to  learn  discriminations  among  items 
within  a class  means  that  a greater  context  of  meaningfulness  can  be  organized 
psychologically. 

Memory  search.  The  strategy  of  searching  memory  for  content  not  immediately 
"recalled  consists  of  selecting  associative  keys  and  building  a context  that  is 
likely  to  elicit  the  content.  Describe  over  the  telephone  how  to  use  the  fire 
extinguisher  that  is  mounted  on  the  wall  in  the  office  corridor.  How  is  it 
done?  You  once  knew  how  to  do  it,  but  that  was  years  ago.  You  have  read  the 
instructions  for  use,  but  have  never  used  one.  You  visualize  the  extinguisher; 
this  visualization  suggests  the  ring  mounted  on  its  top.  In  turn,  this  calls 
up  the  instruction:  invert  the  extinguisher.  You  visualize  it  in  an  inverted 

position  and  this  may  suggest  the  final  procedural  step:  bang  it  on  the  floor 

upside  down  to  break  the  internal  container  that  releases  the  chemical  agent. 

Your  present  visualization  helps  unlock  the  visualization  of  the  action  you  made 
when  you  first  read  the  instructions  on  the  brass  plate. 

What  did  the  instructor  tell  you  to  do  if  a side  wheel  tire  blew  out  when  you 
touch  the  runway?  (This  recall  route  must  be  rapid! ) The  memory  may  be  of  a 
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set  of  verbal  instructions,  or  it  may  be  a visualization  of  the  principle  at 
the  time  of  instruction,  or  a combination  of  both.  But  the  process  consists 
of  one  or  more  convergent  keys  to  the  content.  The  recall  process  may  occur 
in  less  than  a second,  or  may  be  built  up  over  months  of  moving  through  a succes- 
sion of  contexts  as  in  psychoanalysis.  The  context  that  can  be  recalled  first 
is  the  partial  key  to  recall  of  the  next  more  specific  context,  and  the  summing 
of  these  contexts  cumulates  through  a series  of  recognitions  towards  the  object 
of  the  search. 

In  terms  of  background  learning,  the  better  organized  the  mnemonic  pathways, 
the  more  rapid  and  reliable  the  recall.  The  mnemonic  organization  may  consist 
of  hierarchic  groupings  and  subgroupings  of  subject  matter  (potentially  charac- 
teristic of  formal  and  technical  subject  matter)  or  it  may  be  "coordi natively 
indexed"  by  classes  of  environment  and  situation  (such  as  in  personal  exper- 
iences). Coordinate  indexing  is  a librarian's  expression  for  using  a number  of 
descriptors  which,  taken  together,  describe  the  subject  matter  target  of  the 
search. 

Strategy  in  searching  deep  memory  also  consists  of  trying  to  guess  the  code  in 
which  the  content  is  likely  to  be  represented  --  verbal,  pictorial,  diagrammatic, 
kinesthetic,  abstract  conceptual.  The  code  may  not  be  understood  explicitly  in 
these  terms,  but  rather  in  situational  terms.  Another  facet  of  deep  memory 
search  strategy  is  to  guess  at  the  likely  environmental  setting  in  which  the 
learning  experience  occurred.  Examples  described  earlier  are  relevant. 

The  accuracy  of  the  recalled  content  is,  of  course,  subject  to  a variety  of 
distorting  influences  well  known  in  the  psychological  literature.  These  dis- 
tortions may  arise  from  conditions  at  the  time  of  the  experience  (stereotype, 
selective  attention,  inappropriate  set,  perceptual  defense  mechanisms)  or  from 
intervening  experiences  which  are  assimilated  into  old  memory  contexts  and  thus 
alter  them  perhaps  irrevocably.  A given  memory  content  almost  certainly  inter- 
acts with  the  conditions  of  attempted  recall,  and  is  potentially  distorted  by 
that  interaction. 

The  logical  implication  of  this  proposition  is  that  a recall  route  should  not 
be  attempted  which  takes  Use  search  through  somewhat  similar  but  different 
classes  of  content.  Rather,  the  scrategy  should  consist  of  an  attempt  to  work 
with  mnemonic  content  that  is  likely  to  be  unique  context  to  the  target  con- 
tent to  be  recalled.  Por  example,  in  attempting  to  recall  the  street  number 
of  the  house  lived  in  as  a boy,  it  would  seem  poor  strategy  to  work  backward 
from  more  recent  addresses  to  the  address  in  question.  Interference  effects 
among  the  various  addresses  are  virtually  certain.  A better  strategy  to  apply 
in  attempting  to  evoke  the  old  house  number  would  seem  to  be  that  of  picturing 
the  house  in  question  and  the  experience  contexts  associated  solely  with  it. 

(At  least  one  subject  has  been  able  to  recall  extraordinary  details  from  deep 
memory  by  following  this  method  of  what  might  be  called  "experience  reconstruc- 
tion." Practice  at  it  seem£  to  improve  the  capability  in  a general  sense.  It 
is  as  if  one  can  be  trained  to-be  more  sensitive  to  delicate  internal  cues.) 

Additional  techniques  and  strategies  for  searching  and  uncovering  deep  memory 
content  remain  to  be  discovered  and  elucidated  as  procedures.  Although  many 
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of  the  examples  cited  above  have  a personal  flavor,  the  same  principles  most 
certainly  apply  to  content  relevant  to  work  contexts. 

When  a search  strategy  is  successful  in  eliciting  a memory  content,  that  content 
tends  to  become  more  accessible  to  future  searches  — as  if  it  were  brought  from 
background  to  foreground,  so  that  savings  in  search  effort  per  trail  are  very 
large.  This  phenomenon  should  be  a useful  guide  to  the  design  of  refresher 
training  of  "forgotten"  or  partially  forgotten  procedures,  facts,  and  principles. 
For  efficiency  in  relearning,  do  the  relearning  from  the  same  textbook  used  in 
the  original  learning. 

Closely  Related  Topics 

Storage  in  long-term  memory  is  closely  akin  to  the  following  other  terms  in  our 
information  processing  vocabulary:  Identify,  Interpret,  and,  of  course,  Short- 

Term  Memory. 
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